{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SplitRec：在隐语拆分学习中使用流水线并行\n",
    "\n",
    "> 以下代码仅作为示例，请勿在生产环境直接使用。\n",
    "\n",
    "> 本示例基于基于“拆分学习：银行营销”教程制作，建议先观看那个教程。\n",
    "\n",
    "在拆分学习中，由于模型被拆分在多个设备当中，进行训练的时候，各方需要对中间结果和梯度进行多次传输，计算和网络通信存在大量 idle 时间，我们在隐语中参考论文[《PipeLearn: Pipeline Parallelism for \n",
    "Collaborative Machine Learnin》](https://arxiv.org/pdf/2302.12803.pdf)，利用隐语底座 RayFed 的任务调度能力，实现了流水线并行，使得计算和通信能够交叠隐藏部分计算时间，提高资源利用率。由于计算和通信的并发执行，使用流水线并行可能会带来模型准确性上的损失，用户可以根据实际场景平衡性能和精度。\n",
    "\n",
    "下面我们通过一个例子来看一下如何使用在隐语拆分学习中使用流水线并行。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 环境设置\n",
    "首先，我们在 secretflow 环境中创造 2 个实体 alice 和 bob。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-09-26 19:49:23,600\tINFO worker.py:1538 -- Started a local Ray instance.\n"
     ]
    }
   ],
   "source": [
    "import secretflow as sf\n",
    "\n",
    "sf.shutdown()\n",
    "sf.init(['alice', 'bob'], address='local')\n",
    "alice, bob = sf.PYU('alice'), sf.PYU('bob')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准备数据\n",
    "接下来我们准备要学习的数据。\n",
    "\n",
    "我们使用“拆分学习：银行营销”中的数据准备和处理方法，下载银行营销数据集并进行处理。alice 和 bob 的角色和之前的教程完全相同："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from secretflow.utils.simulation.datasets import load_bank_marketing\n",
    "from secretflow.preprocessing.scaler import MinMaxScaler\n",
    "from secretflow.preprocessing.encoder import LabelEncoder\n",
    "from secretflow.data.split import train_test_split\n",
    "\n",
    "random_state = 1234\n",
    "\n",
    "data = load_bank_marketing(parts={alice: (0, 4), bob: (4, 16)}, axis=1)\n",
    "label = load_bank_marketing(parts={alice: (16, 17)}, axis=1)\n",
    "\n",
    "encoder = LabelEncoder()\n",
    "data['job'] = encoder.fit_transform(data['job'])\n",
    "data['marital'] = encoder.fit_transform(data['marital'])\n",
    "data['education'] = encoder.fit_transform(data['education'])\n",
    "data['default'] = encoder.fit_transform(data['default'])\n",
    "data['housing'] = encoder.fit_transform(data['housing'])\n",
    "data['loan'] = encoder.fit_transform(data['loan'])\n",
    "data['contact'] = encoder.fit_transform(data['contact'])\n",
    "data['poutcome'] = encoder.fit_transform(data['poutcome'])\n",
    "data['month'] = encoder.fit_transform(data['month'])\n",
    "label = encoder.fit_transform(label)\n",
    "\n",
    "scaler = MinMaxScaler()\n",
    "data = scaler.fit_transform(data)\n",
    "\n",
    "train_data, test_data = train_test_split(\n",
    "    data, train_size=0.8, random_state=random_state\n",
    ")\n",
    "train_label, test_label = train_test_split(\n",
    "    label, train_size=0.8, random_state=random_state\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 定义模型结构\n",
    "\n",
    "接下来我们创建联邦模型，同样地，我们使用“拆分学习：银行营销”中的建模，构建出 base_model 和 fuse_model，然后就可以定义 SLModel 用于训练："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_base_model(input_dim, output_dim, name='base_model'):\n",
    "    # Create model\n",
    "    def create_model():\n",
    "        from tensorflow import keras\n",
    "        from tensorflow.keras import layers\n",
    "        import tensorflow as tf\n",
    "\n",
    "        model = keras.Sequential(\n",
    "            [\n",
    "                keras.Input(shape=input_dim),\n",
    "                layers.Dense(100, activation=\"relu\"),\n",
    "                layers.Dense(output_dim, activation=\"relu\"),\n",
    "            ]\n",
    "        )\n",
    "        # Compile model\n",
    "        model.summary()\n",
    "        model.compile(\n",
    "            loss='binary_crossentropy',\n",
    "            optimizer='adam',\n",
    "            metrics=[\"accuracy\", tf.keras.metrics.AUC()],\n",
    "        )\n",
    "        return model\n",
    "\n",
    "    return create_model\n",
    "\n",
    "\n",
    "# prepare model\n",
    "hidden_size = 64\n",
    "\n",
    "model_base_alice = create_base_model(4, hidden_size)\n",
    "model_base_bob = create_base_model(12, hidden_size)\n",
    "\n",
    "\n",
    "def create_fuse_model(input_dim, output_dim, party_nums, name='fuse_model'):\n",
    "    def create_model():\n",
    "        from tensorflow import keras\n",
    "        from tensorflow.keras import layers\n",
    "        import tensorflow as tf\n",
    "\n",
    "        # input\n",
    "        input_layers = []\n",
    "        for i in range(party_nums):\n",
    "            input_layers.append(\n",
    "                keras.Input(\n",
    "                    input_dim,\n",
    "                )\n",
    "            )\n",
    "\n",
    "        merged_layer = layers.concatenate(input_layers)\n",
    "        fuse_layer = layers.Dense(64, activation='relu')(merged_layer)\n",
    "        output = layers.Dense(output_dim, activation='sigmoid')(fuse_layer)\n",
    "\n",
    "        model = keras.Model(inputs=input_layers, outputs=output)\n",
    "        model.summary()\n",
    "\n",
    "        model.compile(\n",
    "            loss='binary_crossentropy',\n",
    "            optimizer='adam',\n",
    "            metrics=[\"accuracy\", tf.keras.metrics.AUC()],\n",
    "        )\n",
    "        return model\n",
    "\n",
    "    return create_model\n",
    "\n",
    "\n",
    "model_fuse = create_fuse_model(input_dim=hidden_size, party_nums=2, output_dim=1)\n",
    "\n",
    "base_model_dict = {alice: model_base_alice, bob: model_base_bob}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 定义 SLModel\n",
    "这里如果使用流水线并行，设置 strategy = 'pipline'，并设置参数 pipeline_size，pipeline_size 增大并发程度会增大，但达到一定阈值，当一方的计算或网络被打满，性能将不会再有提升，通常 pipeline_size 设为 2-4。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(_run pid=1817588)\u001b[0m /home/ssd2/zhaocaibei/miniconda3/envs/jupyter/lib/python3.8/site-packages/sklearn/base.py:443: UserWarning: X has feature names, but MinMaxScaler was fitted without feature names\n",
      "\u001b[2m\u001b[36m(_run pid=1817588)\u001b[0m   warnings.warn(\n",
      "\u001b[2m\u001b[36m(_run pid=1817882)\u001b[0m /home/ssd2/zhaocaibei/miniconda3/envs/jupyter/lib/python3.8/site-packages/sklearn/base.py:443: UserWarning: X has feature names, but MinMaxScaler was fitted without feature names\n",
      "\u001b[2m\u001b[36m(_run pid=1817882)\u001b[0m   warnings.warn(\n",
      "INFO:root:Create proxy actor <class 'secretflow.ml.nn.sl.backend.tensorflow.sl_base.PYUSLTFModel'> with party alice.\n",
      "INFO:root:Create proxy actor <class 'secretflow.ml.nn.sl.backend.tensorflow.sl_base.PYUSLTFModel'> with party bob.\n",
      "INFO:root:Create proxy actor <class 'secretflow.ml.nn.sl.backend.tensorflow.strategy.pipeline.PYUPipelineTFModel'> with party alice.\n",
      "INFO:root:Create proxy actor <class 'secretflow.ml.nn.sl.backend.tensorflow.strategy.pipeline.PYUPipelineTFModel'> with party bob.\n"
     ]
    }
   ],
   "source": [
    "from secretflow.ml.nn import SLModel\n",
    "\n",
    "sl_model_origin = SLModel(\n",
    "    base_model_dict=base_model_dict,\n",
    "    device_y=alice,\n",
    "    model_fuse=model_fuse,\n",
    ")\n",
    "\n",
    "sl_model_pipeline = SLModel(\n",
    "    base_model_dict=base_model_dict,\n",
    "    device_y=alice,\n",
    "    model_fuse=model_fuse,\n",
    "    strategy='pipeline',\n",
    "    pipeline_size=2,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 开始训练\n",
    "\n",
    "我们分别对没有使用通讯压缩的模型和使用了量化压缩的模型进行训练，并把训练轮次拉高到40轮，看看效果如何。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:SL Train Params: {'x': VDataFrame(partitions={PYURuntime(alice): <secretflow.data.partition.pandas.partition.PdPartition object at 0x7f13d0aa04c0>, PYURuntime(bob): <secretflow.data.partition.pandas.partition.PdPartition object at 0x7f136874bcd0>}, aligned=True), 'y': VDataFrame(partitions={PYURuntime(alice): <secretflow.data.partition.pandas.partition.PdPartition object at 0x7f13d0ac8430>}, aligned=True), 'batch_size': 128, 'epochs': 40, 'verbose': 1, 'callbacks': None, 'validation_data': (VDataFrame(partitions={PYURuntime(alice): <secretflow.data.partition.pandas.partition.PdPartition object at 0x7f136874bc10>, PYURuntime(bob): <secretflow.data.partition.pandas.partition.PdPartition object at 0x7f136874b400>}, aligned=True), VDataFrame(partitions={PYURuntime(alice): <secretflow.data.partition.pandas.partition.PdPartition object at 0x7f136874be20>}, aligned=True)), 'shuffle': True, 'sample_weight': None, 'validation_freq': 1, 'dp_spent_step_freq': None, 'dataset_builder': None, 'audit_log_params': {}, 'random_seed': 5731, 'audit_log_dir': None, 'self': <secretflow.ml.nn.sl.sl_model.SLModel object at 0x7f1368738df0>}\n",
      "\u001b[2m\u001b[36m(pid=1825002)\u001b[0m 2023-09-26 19:49:33.620129: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst\n",
      "\u001b[2m\u001b[36m(pid=1825080)\u001b[0m 2023-09-26 19:49:33.727665: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst\n",
      "\u001b[2m\u001b[36m(pid=1825145)\u001b[0m 2023-09-26 19:49:34.036428: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst\n",
      "\u001b[2m\u001b[36m(pid=1825269)\u001b[0m 2023-09-26 19:49:36.259301: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst\n",
      "\u001b[2m\u001b[36m(pid=1825002)\u001b[0m 2023-09-26 19:49:36.484667: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst\n",
      "\u001b[2m\u001b[36m(pid=1825002)\u001b[0m 2023-09-26 19:49:36.484792: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst\n",
      "\u001b[2m\u001b[36m(pid=1825002)\u001b[0m 2023-09-26 19:49:36.484806: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n",
      "\u001b[2m\u001b[36m(pid=1825080)\u001b[0m 2023-09-26 19:49:36.757167: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst\n",
      "\u001b[2m\u001b[36m(pid=1825080)\u001b[0m 2023-09-26 19:49:36.757286: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst\n",
      "\u001b[2m\u001b[36m(pid=1825080)\u001b[0m 2023-09-26 19:49:36.757300: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n",
      "\u001b[2m\u001b[36m(pid=1825145)\u001b[0m 2023-09-26 19:49:36.855295: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst\n",
      "\u001b[2m\u001b[36m(pid=1825145)\u001b[0m 2023-09-26 19:49:36.855437: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst\n",
      "\u001b[2m\u001b[36m(pid=1825145)\u001b[0m 2023-09-26 19:49:36.855455: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n",
      "\u001b[2m\u001b[36m(pid=1825269)\u001b[0m 2023-09-26 19:49:37.154045: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst\n",
      "\u001b[2m\u001b[36m(pid=1825269)\u001b[0m 2023-09-26 19:49:37.154148: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst\n",
      "\u001b[2m\u001b[36m(pid=1825269)\u001b[0m 2023-09-26 19:49:37.154161: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m 2023-09-26 19:49:38.824645: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m 2023-09-26 19:49:38.824774: W tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:265] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825080)\u001b[0m 2023-09-26 19:49:38.909952: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825080)\u001b[0m 2023-09-26 19:49:38.909992: W tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:265] failed call to cuInit: UNKNOWN ERROR (303)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m Model: \"sequential\"\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m _________________________________________________________________\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m  Layer (type)                Output Shape              Param #   \n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m =================================================================\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m  dense (Dense)               (None, 100)               500       \n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m                                                                  \n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m  dense_1 (Dense)             (None, 64)                6464      \n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m                                                                  \n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m =================================================================\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m Total params: 6,964\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m Trainable params: 6,964\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m Non-trainable params: 0\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m _________________________________________________________________\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m Model: \"model\"\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m __________________________________________________________________________________________________\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m  Layer (type)                   Output Shape         Param #     Connected to                     \n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m ==================================================================================================\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m  input_2 (InputLayer)           [(None, 64)]         0           []                               \n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m                                                                                                   \n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m  input_3 (InputLayer)           [(None, 64)]         0           []                               \n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m                                                                                                   \n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m  concatenate (Concatenate)      (None, 128)          0           ['input_2[0][0]',                \n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m                                                                   'input_3[0][0]']                \n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m                                                                                                   \n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m  dense_2 (Dense)                (None, 64)           8256        ['concatenate[0][0]']            \n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m                                                                                                   \n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m  dense_3 (Dense)                (None, 1)            65          ['dense_2[0][0]']                \n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m                                                                                                   \n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m ==================================================================================================\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m Total params: 8,321\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m Trainable params: 8,321\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m Non-trainable params: 0\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825002)\u001b[0m __________________________________________________________________________________________________\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825080)\u001b[0m Model: \"sequential\"\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825080)\u001b[0m _________________________________________________________________\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825080)\u001b[0m  Layer (type)                Output Shape              Param #   \n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825080)\u001b[0m =================================================================\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825080)\u001b[0m  dense (Dense)               (None, 100)               1300      \n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825080)\u001b[0m                                                                  \n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825080)\u001b[0m  dense_1 (Dense)             (None, 64)                6464      \n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825080)\u001b[0m                                                                  \n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825080)\u001b[0m =================================================================\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825080)\u001b[0m Total params: 7,764\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825080)\u001b[0m Trainable params: 7,764\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825080)\u001b[0m Non-trainable params: 0\n",
      "\u001b[2m\u001b[36m(PYUSLTFModel pid=1825080)\u001b[0m _________________________________________________________________\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/29 [00:00<?, ?it/s]\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m 2023-09-26 19:49:39.329376: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst\n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m 2023-09-26 19:49:39.329425: W tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:265] failed call to cuInit: UNKNOWN ERROR (303)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m Model: \"sequential\"\n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m _________________________________________________________________\n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m  Layer (type)                Output Shape              Param #   \n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m =================================================================\n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m  dense (Dense)               (None, 100)               500       \n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m                                                                  \n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m  dense_1 (Dense)             (None, 64)                6464      \n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m                                                                  \n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m =================================================================\n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m Total params: 6,964\n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m Trainable params: 6,964\n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m Non-trainable params: 0\n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m _________________________________________________________________\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825269)\u001b[0m 2023-09-26 19:49:39.538987: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst\n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825269)\u001b[0m 2023-09-26 19:49:39.539186: W tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:265] failed call to cuInit: UNKNOWN ERROR (303)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m Model: \"model\"\n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m __________________________________________________________________________________________________\n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m  Layer (type)                   Output Shape         Param #     Connected to                     \n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m ==================================================================================================\n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m  input_2 (InputLayer)           [(None, 64)]         0           []                               \n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m                                                                                                   \n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m  input_3 (InputLayer)           [(None, 64)]         0           []                               \n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m                                                                                                   \n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m  concatenate (Concatenate)      (None, 128)          0           ['input_2[0][0]',                \n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m                                                                   'input_3[0][0]']                \n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m                                                                                                   \n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m  dense_2 (Dense)                (None, 64)           8256        ['concatenate[0][0]']            \n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m                                                                                                   \n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m  dense_3 (Dense)                (None, 1)            65          ['dense_2[0][0]']                \n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m                                                                                                   \n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m ==================================================================================================\n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m Total params: 8,321\n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m Trainable params: 8,321\n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m Non-trainable params: 0\n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825145)\u001b[0m __________________________________________________________________________________________________\n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825269)\u001b[0m Model: \"sequential\"\n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825269)\u001b[0m _________________________________________________________________\n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825269)\u001b[0m  Layer (type)                Output Shape              Param #   \n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825269)\u001b[0m =================================================================\n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825269)\u001b[0m  dense (Dense)               (None, 100)               1300      \n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825269)\u001b[0m                                                                  \n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825269)\u001b[0m  dense_1 (Dense)             (None, 64)                6464      \n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825269)\u001b[0m                                                                  \n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825269)\u001b[0m =================================================================\n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825269)\u001b[0m Total params: 7,764\n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825269)\u001b[0m Trainable params: 7,764\n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825269)\u001b[0m Non-trainable params: 0\n",
      "\u001b[2m\u001b[36m(PYUPipelineTFModel pid=1825269)\u001b[0m _________________________________________________________________\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  7%|▋         | 2/29 [00:02<00:38,  1.43s/it]\u001b[2m\u001b[36m(_run pid=1817588)\u001b[0m 2023-09-26 19:49:42.338308: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst\n",
      "\u001b[2m\u001b[36m(_run pid=1817588)\u001b[0m 2023-09-26 19:49:43.232958: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst\n",
      "\u001b[2m\u001b[36m(_run pid=1817588)\u001b[0m 2023-09-26 19:49:43.233049: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst\n",
      "\u001b[2m\u001b[36m(_run pid=1817588)\u001b[0m 2023-09-26 19:49:43.233059: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n",
      "\u001b[2m\u001b[36m(_run pid=1817588)\u001b[0m 2023-09-26 19:49:45.244641: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst\n",
      "\u001b[2m\u001b[36m(_run pid=1817588)\u001b[0m 2023-09-26 19:49:45.244680: W tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:265] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "100%|██████████| 29/29 [00:08<00:00,  3.54it/s, epoch: 1/40 -  train_loss:0.44887787103652954  train_accuracy:0.850215494632721  train_auc_1:0.5317299365997314  val_loss:0.39494723081588745  val_accuracy:0.8729282021522522  val_auc_1:0.5657897591590881 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 16.09it/s, epoch: 2/40 -  train_loss:0.3432118892669678  train_accuracy:0.8857954740524292  train_auc_1:0.6473174095153809  val_loss:0.363627165555954  val_accuracy:0.8729282021522522  val_auc_1:0.6689268350601196 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 26.94it/s, epoch: 3/40 -  train_loss:0.32648009061813354  train_accuracy:0.8863146305084229  train_auc_1:0.7191672921180725  val_loss:0.35098856687545776  val_accuracy:0.8729282021522522  val_auc_1:0.7191359400749207 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 26.13it/s, epoch: 4/40 -  train_loss:0.31278952956199646  train_accuracy:0.8833512663841248  train_auc_1:0.7800465226173401  val_loss:0.34081292152404785  val_accuracy:0.8729282021522522  val_auc_1:0.7596642971038818 ]\n",
      "  0%|          | 0/29 [00:00<?, ?it/s]\u001b[2m\u001b[36m(_run pid=1817740)\u001b[0m 2023-09-26 19:49:51.567820: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst\n",
      "\u001b[2m\u001b[36m(_run pid=1817740)\u001b[0m 2023-09-26 19:49:52.431220: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst\n",
      "\u001b[2m\u001b[36m(_run pid=1817740)\u001b[0m 2023-09-26 19:49:52.431321: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst\n",
      "\u001b[2m\u001b[36m(_run pid=1817740)\u001b[0m 2023-09-26 19:49:52.431334: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n",
      "\u001b[2m\u001b[36m(_run pid=1817740)\u001b[0m 2023-09-26 19:49:54.400120: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst\n",
      "\u001b[2m\u001b[36m(_run pid=1817740)\u001b[0m 2023-09-26 19:49:54.400158: W tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:265] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "100%|██████████| 29/29 [00:04<00:00,  6.83it/s, epoch: 5/40 -  train_loss:0.29879459738731384  train_accuracy:0.8844026327133179  train_auc_1:0.8006787300109863  val_loss:0.32317325472831726  val_accuracy:0.8729282021522522  val_auc_1:0.801051139831543 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 26.10it/s, epoch: 6/40 -  train_loss:0.29146328568458557  train_accuracy:0.875  train_auc_1:0.8463853597640991  val_loss:0.3064562678337097  val_accuracy:0.8696132302284241  val_auc_1:0.8238029479980469 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 26.24it/s, epoch: 7/40 -  train_loss:0.24985690414905548  train_accuracy:0.8960176706314087  train_auc_1:0.8685527443885803  val_loss:0.30176323652267456  val_accuracy:0.8718231916427612  val_auc_1:0.839279055595398 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 26.07it/s, epoch: 8/40 -  train_loss:0.25740572810173035  train_accuracy:0.8872159123420715  train_auc_1:0.8768747448921204  val_loss:0.2859904170036316  val_accuracy:0.8806629776954651  val_auc_1:0.8433902263641357 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.53it/s, epoch: 9/40 -  train_loss:0.24906550347805023  train_accuracy:0.8995150923728943  train_auc_1:0.8653609156608582  val_loss:0.2812442481517792  val_accuracy:0.8828729391098022  val_auc_1:0.8512052893638611 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.23it/s, epoch: 10/40 -  train_loss:0.2445402294397354  train_accuracy:0.892699122428894  train_auc_1:0.8836410641670227  val_loss:0.2827773094177246  val_accuracy:0.8773480653762817  val_auc_1:0.8469785451889038 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 26.02it/s, epoch: 11/40 -  train_loss:0.2518855035305023  train_accuracy:0.8954645991325378  train_auc_1:0.8761004209518433  val_loss:0.295387327671051  val_accuracy:0.8784530162811279  val_auc_1:0.8485745787620544 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.55it/s, epoch: 12/40 -  train_loss:0.22354044020175934  train_accuracy:0.9102909564971924  train_auc_1:0.8964704275131226  val_loss:0.30353987216949463  val_accuracy:0.8795580267906189  val_auc_1:0.8534507155418396 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 28.33it/s, epoch: 13/40 -  train_loss:0.22443315386772156  train_accuracy:0.9079092741012573  train_auc_1:0.8922196626663208  val_loss:0.2777591645717621  val_accuracy:0.8795580267906189  val_auc_1:0.8531700372695923 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 25.85it/s, epoch: 14/40 -  train_loss:0.21603752672672272  train_accuracy:0.9125000238418579  train_auc_1:0.9022694230079651  val_loss:0.2857709228992462  val_accuracy:0.8817679286003113  val_auc_1:0.8522068858146667 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 26.38it/s, epoch: 15/40 -  train_loss:0.2281351238489151  train_accuracy:0.9126105904579163  train_auc_1:0.8849776983261108  val_loss:0.27802714705467224  val_accuracy:0.8773480653762817  val_auc_1:0.853461742401123 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.39it/s, epoch: 16/40 -  train_loss:0.2165425419807434  train_accuracy:0.9150568246841431  train_auc_1:0.8966040015220642  val_loss:0.2783280313014984  val_accuracy:0.8828729391098022  val_auc_1:0.8554485440254211 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 28.48it/s, epoch: 17/40 -  train_loss:0.21801750361919403  train_accuracy:0.9081858396530151  train_auc_1:0.8976572155952454  val_loss:0.2761484980583191  val_accuracy:0.8784530162811279  val_auc_1:0.8577655553817749 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 28.16it/s, epoch: 18/40 -  train_loss:0.2221526950597763  train_accuracy:0.907866358757019  train_auc_1:0.9024293422698975  val_loss:0.2797839343547821  val_accuracy:0.8828729391098022  val_auc_1:0.8589048385620117 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.62it/s, epoch: 19/40 -  train_loss:0.2285359650850296  train_accuracy:0.9059734344482422  train_auc_1:0.8901480436325073  val_loss:0.27858439087867737  val_accuracy:0.8828729391098022  val_auc_1:0.860638439655304 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.14it/s, epoch: 20/40 -  train_loss:0.2182772159576416  train_accuracy:0.915099561214447  train_auc_1:0.9078006744384766  val_loss:0.291841983795166  val_accuracy:0.8751381039619446  val_auc_1:0.8562520742416382 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.61it/s, epoch: 21/40 -  train_loss:0.20946133136749268  train_accuracy:0.9164772629737854  train_auc_1:0.9116370677947998  val_loss:0.2933138310909271  val_accuracy:0.8828729391098022  val_auc_1:0.8598623275756836 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 26.59it/s, epoch: 22/40 -  train_loss:0.23493120074272156  train_accuracy:0.9054203629493713  train_auc_1:0.8970139026641846  val_loss:0.27499568462371826  val_accuracy:0.8773480653762817  val_auc_1:0.8625756502151489 ]\n",
      "100%|██████████| 29/29 [00:00<00:00, 29.05it/s, epoch: 23/40 -  train_loss:0.21671472489833832  train_accuracy:0.9101216793060303  train_auc_1:0.9046225547790527  val_loss:0.2828799784183502  val_accuracy:0.8850829005241394  val_auc_1:0.8609355688095093 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 28.19it/s, epoch: 24/40 -  train_loss:0.22586138546466827  train_accuracy:0.9110991358757019  train_auc_1:0.90799880027771  val_loss:0.28323644399642944  val_accuracy:0.8850829005241394  val_auc_1:0.8611777424812317 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 28.10it/s, epoch: 25/40 -  train_loss:0.21121767163276672  train_accuracy:0.9133522510528564  train_auc_1:0.9088505506515503  val_loss:0.27677592635154724  val_accuracy:0.8861878514289856  val_auc_1:0.860908031463623 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.28it/s, epoch: 26/40 -  train_loss:0.20616813004016876  train_accuracy:0.9184659123420715  train_auc_1:0.908971905708313  val_loss:0.2886063754558563  val_accuracy:0.8828729391098022  val_auc_1:0.8584039807319641 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 25.97it/s, epoch: 27/40 -  train_loss:0.24042247235774994  train_accuracy:0.8976293206214905  train_auc_1:0.8970732688903809  val_loss:0.2905130684375763  val_accuracy:0.8817679286003113  val_auc_1:0.8628398180007935 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 28.27it/s, epoch: 28/40 -  train_loss:0.2106049805879593  train_accuracy:0.9131637215614319  train_auc_1:0.9120412468910217  val_loss:0.2760363817214966  val_accuracy:0.8883978128433228  val_auc_1:0.8631425499916077 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.53it/s, epoch: 29/40 -  train_loss:0.19771815836429596  train_accuracy:0.9181034564971924  train_auc_1:0.9175819158554077  val_loss:0.2815971374511719  val_accuracy:0.8872928023338318  val_auc_1:0.8594056367874146 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.45it/s, epoch: 30/40 -  train_loss:0.21977882087230682  train_accuracy:0.9065265655517578  train_auc_1:0.908697247505188  val_loss:0.2787911891937256  val_accuracy:0.8806629776954651  val_auc_1:0.8623830080032349 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 26.70it/s, epoch: 31/40 -  train_loss:0.2060454785823822  train_accuracy:0.9121767282485962  train_auc_1:0.909172534942627  val_loss:0.29584282636642456  val_accuracy:0.8806629776954651  val_auc_1:0.8634011149406433 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.61it/s, epoch: 32/40 -  train_loss:0.20517688989639282  train_accuracy:0.9191810488700867  train_auc_1:0.907102108001709  val_loss:0.28416600823402405  val_accuracy:0.8839778900146484  val_auc_1:0.8632196187973022 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.90it/s, epoch: 33/40 -  train_loss:0.21313920617103577  train_accuracy:0.9112278819084167  train_auc_1:0.9166741371154785  val_loss:0.2824288308620453  val_accuracy:0.8817679286003113  val_auc_1:0.8626307249069214 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.56it/s, epoch: 34/40 -  train_loss:0.20695945620536804  train_accuracy:0.9164823293685913  train_auc_1:0.9163408279418945  val_loss:0.2820662260055542  val_accuracy:0.8861878514289856  val_auc_1:0.8618767261505127 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.02it/s, epoch: 35/40 -  train_loss:0.2142862230539322  train_accuracy:0.9129849076271057  train_auc_1:0.9152473211288452  val_loss:0.2856462597846985  val_accuracy:0.8806629776954651  val_auc_1:0.8672426342964172 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.22it/s, epoch: 36/40 -  train_loss:0.18980328738689423  train_accuracy:0.92578125  train_auc_1:0.9262045621871948  val_loss:0.2957724332809448  val_accuracy:0.8850829005241394  val_auc_1:0.855277955532074 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.44it/s, epoch: 37/40 -  train_loss:0.2022796869277954  train_accuracy:0.9156526327133179  train_auc_1:0.9175050854682922  val_loss:0.28874075412750244  val_accuracy:0.8828729391098022  val_auc_1:0.8578150272369385 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 26.17it/s, epoch: 38/40 -  train_loss:0.20826201140880585  train_accuracy:0.917588472366333  train_auc_1:0.9161174893379211  val_loss:0.28712138533592224  val_accuracy:0.889502763748169  val_auc_1:0.8563125729560852 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 28.39it/s, epoch: 39/40 -  train_loss:0.20791961252689362  train_accuracy:0.9195243120193481  train_auc_1:0.9068150520324707  val_loss:0.28275591135025024  val_accuracy:0.8828729391098022  val_auc_1:0.8621078133583069 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 26.34it/s, epoch: 40/40 -  train_loss:0.2018997073173523  train_accuracy:0.9161931872367859  train_auc_1:0.9205420613288879  val_loss:0.2885696589946747  val_accuracy:0.8839778900146484  val_auc_1:0.8595322370529175 ]\n",
      "INFO:root:SL Train Params: {'x': VDataFrame(partitions={PYURuntime(alice): <secretflow.data.partition.pandas.partition.PdPartition object at 0x7f13d0aa04c0>, PYURuntime(bob): <secretflow.data.partition.pandas.partition.PdPartition object at 0x7f136874bcd0>}, aligned=True), 'y': VDataFrame(partitions={PYURuntime(alice): <secretflow.data.partition.pandas.partition.PdPartition object at 0x7f13d0ac8430>}, aligned=True), 'batch_size': 128, 'epochs': 40, 'verbose': 1, 'callbacks': None, 'validation_data': (VDataFrame(partitions={PYURuntime(alice): <secretflow.data.partition.pandas.partition.PdPartition object at 0x7f136874bc10>, PYURuntime(bob): <secretflow.data.partition.pandas.partition.PdPartition object at 0x7f136874b400>}, aligned=True), VDataFrame(partitions={PYURuntime(alice): <secretflow.data.partition.pandas.partition.PdPartition object at 0x7f136874be20>}, aligned=True)), 'shuffle': True, 'sample_weight': None, 'validation_freq': 1, 'dp_spent_step_freq': None, 'dataset_builder': None, 'audit_log_params': {}, 'random_seed': 57815, 'audit_log_dir': None, 'self': <secretflow.ml.nn.sl.sl_model.SLModel object at 0x7f1368738e20>}\n",
      "100%|██████████| 29/29 [00:03<00:00,  9.21it/s, epoch: 1/40 -  train_loss:0.4289693236351013  train_accuracy:0.841261088848114  train_auc_1:0.5522745847702026  val_loss:0.4278267025947571  val_accuracy:0.8729282021522522  val_auc_1:0.596114456653595 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.58it/s, epoch: 2/40 -  train_loss:0.3457517921924591  train_accuracy:0.8894886374473572  train_auc_1:0.5915822982788086  val_loss:0.36710259318351746  val_accuracy:0.8729282021522522  val_auc_1:0.6552668809890747 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.34it/s, epoch: 3/40 -  train_loss:0.33160045742988586  train_accuracy:0.8857758641242981  train_auc_1:0.6996316909790039  val_loss:0.3565421402454376  val_accuracy:0.8729282021522522  val_auc_1:0.6939350962638855 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 26.56it/s, epoch: 4/40 -  train_loss:0.3203859031200409  train_accuracy:0.8879978060722351  train_auc_1:0.7228385210037231  val_loss:0.34700706601142883  val_accuracy:0.8729282021522522  val_auc_1:0.7337919473648071 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.51it/s, epoch: 5/40 -  train_loss:0.29412248730659485  train_accuracy:0.8941271305084229  train_auc_1:0.7773510217666626  val_loss:0.3330070972442627  val_accuracy:0.8729282021522522  val_auc_1:0.7747275233268738 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 26.64it/s, epoch: 6/40 -  train_loss:0.284542053937912  train_accuracy:0.8840909004211426  train_auc_1:0.8406177759170532  val_loss:0.3239462077617645  val_accuracy:0.8729282021522522  val_auc_1:0.8016015291213989 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 28.29it/s, epoch: 7/40 -  train_loss:0.2678506672382355  train_accuracy:0.8896570801734924  train_auc_1:0.8536190986633301  val_loss:0.2891094386577606  val_accuracy:0.8773480653762817  val_auc_1:0.8471270799636841 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.18it/s, epoch: 8/40 -  train_loss:0.24808000028133392  train_accuracy:0.9051136374473572  train_auc_1:0.8591896891593933  val_loss:0.29273611307144165  val_accuracy:0.8718231916427612  val_auc_1:0.8348816633224487 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 26.80it/s, epoch: 9/40 -  train_loss:0.252238392829895  train_accuracy:0.8917025923728943  train_auc_1:0.8791320323944092  val_loss:0.2933768332004547  val_accuracy:0.8773480653762817  val_auc_1:0.8507869243621826 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.14it/s, epoch: 10/40 -  train_loss:0.24061305820941925  train_accuracy:0.8982300758361816  train_auc_1:0.8796927332878113  val_loss:0.28491833806037903  val_accuracy:0.8784530162811279  val_auc_1:0.843962550163269 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.45it/s, epoch: 11/40 -  train_loss:0.24184739589691162  train_accuracy:0.9035560488700867  train_auc_1:0.8807085752487183  val_loss:0.29128599166870117  val_accuracy:0.8773480653762817  val_auc_1:0.855217456817627 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 28.02it/s, epoch: 12/40 -  train_loss:0.24324959516525269  train_accuracy:0.8994318246841431  train_auc_1:0.8777059316635132  val_loss:0.2829277217388153  val_accuracy:0.8762431144714355  val_auc_1:0.8485085368156433 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 28.26it/s, epoch: 13/40 -  train_loss:0.24976494908332825  train_accuracy:0.9017045497894287  train_auc_1:0.8789190649986267  val_loss:0.3205900490283966  val_accuracy:0.8762431144714355  val_auc_1:0.8470830917358398 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.93it/s, epoch: 14/40 -  train_loss:0.25038978457450867  train_accuracy:0.9027478694915771  train_auc_1:0.87300044298172  val_loss:0.2858302593231201  val_accuracy:0.8817679286003113  val_auc_1:0.8585360646247864 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 28.00it/s, epoch: 15/40 -  train_loss:0.22572343051433563  train_accuracy:0.9065194129943848  train_auc_1:0.8839739561080933  val_loss:0.27690911293029785  val_accuracy:0.8795580267906189  val_auc_1:0.8565822839736938 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 26.32it/s, epoch: 16/40 -  train_loss:0.21374236047267914  train_accuracy:0.9098557829856873  train_auc_1:0.8846680521965027  val_loss:0.28506413102149963  val_accuracy:0.8817679286003113  val_auc_1:0.8479802012443542 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.78it/s, epoch: 17/40 -  train_loss:0.2231924682855606  train_accuracy:0.9081858396530151  train_auc_1:0.8896657228469849  val_loss:0.28099507093429565  val_accuracy:0.8817679286003113  val_auc_1:0.8554320335388184 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.45it/s, epoch: 18/40 -  train_loss:0.21955524384975433  train_accuracy:0.9089439511299133  train_auc_1:0.8989342451095581  val_loss:0.2811129093170166  val_accuracy:0.8795580267906189  val_auc_1:0.8558282852172852 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 28.06it/s, epoch: 19/40 -  train_loss:0.2427460104227066  train_accuracy:0.90625  train_auc_1:0.8821660280227661  val_loss:0.2881392240524292  val_accuracy:0.8762431144714355  val_auc_1:0.853461742401123 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.76it/s, epoch: 20/40 -  train_loss:0.22428719699382782  train_accuracy:0.9095686078071594  train_auc_1:0.8950278759002686  val_loss:0.27665039896965027  val_accuracy:0.8773480653762817  val_auc_1:0.859268069267273 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 28.75it/s, epoch: 21/40 -  train_loss:0.23987431824207306  train_accuracy:0.8982300758361816  train_auc_1:0.8865035772323608  val_loss:0.2787191569805145  val_accuracy:0.8795580267906189  val_auc_1:0.8540340662002563 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 28.38it/s, epoch: 22/40 -  train_loss:0.23535579442977905  train_accuracy:0.9059734344482422  train_auc_1:0.868804931640625  val_loss:0.2837145924568176  val_accuracy:0.8850829005241394  val_auc_1:0.852718710899353 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 26.92it/s, epoch: 23/40 -  train_loss:0.22587361931800842  train_accuracy:0.9102272987365723  train_auc_1:0.9052188396453857  val_loss:0.31282860040664673  val_accuracy:0.8751381039619446  val_auc_1:0.8482223749160767 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 28.01it/s, epoch: 24/40 -  train_loss:0.23454731702804565  train_accuracy:0.9004424810409546  train_auc_1:0.8923315405845642  val_loss:0.2845218777656555  val_accuracy:0.8817679286003113  val_auc_1:0.8535387516021729 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.66it/s, epoch: 25/40 -  train_loss:0.21877069771289825  train_accuracy:0.9135237336158752  train_auc_1:0.8983669877052307  val_loss:0.2817881405353546  val_accuracy:0.8872928023338318  val_auc_1:0.8562355637550354 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 28.56it/s, epoch: 26/40 -  train_loss:0.23317019641399384  train_accuracy:0.9047897458076477  train_auc_1:0.8926054835319519  val_loss:0.2959703505039215  val_accuracy:0.8806629776954651  val_auc_1:0.8549916744232178 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.97it/s, epoch: 27/40 -  train_loss:0.22137722373008728  train_accuracy:0.9076327681541443  train_auc_1:0.9086238145828247  val_loss:0.2853763699531555  val_accuracy:0.8806629776954651  val_auc_1:0.8527958989143372 ]\n",
      "100%|██████████| 29/29 [00:00<00:00, 30.42it/s, epoch: 28/40 -  train_loss:0.21273373067378998  train_accuracy:0.915409505367279  train_auc_1:0.9035125970840454  val_loss:0.2835741639137268  val_accuracy:0.8773480653762817  val_auc_1:0.8541442155838013 ]\n",
      "100%|██████████| 29/29 [00:00<00:00, 30.00it/s, epoch: 29/40 -  train_loss:0.2242117077112198  train_accuracy:0.9054203629493713  train_auc_1:0.9061750173568726  val_loss:0.28261181712150574  val_accuracy:0.8806629776954651  val_auc_1:0.8538470268249512 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 28.50it/s, epoch: 30/40 -  train_loss:0.23390451073646545  train_accuracy:0.90625  train_auc_1:0.8956843614578247  val_loss:0.2910405099391937  val_accuracy:0.8773480653762817  val_auc_1:0.856081485748291 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 28.54it/s, epoch: 31/40 -  train_loss:0.21303458511829376  train_accuracy:0.9097546935081482  train_auc_1:0.9092994928359985  val_loss:0.2907060384750366  val_accuracy:0.8795580267906189  val_auc_1:0.854667067527771 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 28.50it/s, epoch: 32/40 -  train_loss:0.2017047256231308  train_accuracy:0.9213067889213562  train_auc_1:0.9215606451034546  val_loss:0.2792257070541382  val_accuracy:0.8784530162811279  val_auc_1:0.8622398972511292 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.78it/s, epoch: 33/40 -  train_loss:0.21978729963302612  train_accuracy:0.90625  train_auc_1:0.9032835364341736  val_loss:0.2790915071964264  val_accuracy:0.8828729391098022  val_auc_1:0.8596147894859314 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 28.67it/s, epoch: 34/40 -  train_loss:0.2279532253742218  train_accuracy:0.900053858757019  train_auc_1:0.9082023501396179  val_loss:0.28917449712753296  val_accuracy:0.8850829005241394  val_auc_1:0.8572481870651245 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 28.98it/s, epoch: 35/40 -  train_loss:0.20137761533260345  train_accuracy:0.9197198152542114  train_auc_1:0.9088310599327087  val_loss:0.2941972017288208  val_accuracy:0.8839778900146484  val_auc_1:0.860407292842865 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.61it/s, epoch: 36/40 -  train_loss:0.21837779879570007  train_accuracy:0.9113685488700867  train_auc_1:0.9138096570968628  val_loss:0.31745481491088867  val_accuracy:0.8795580267906189  val_auc_1:0.8539350032806396 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.85it/s, epoch: 37/40 -  train_loss:0.22294917702674866  train_accuracy:0.9103982448577881  train_auc_1:0.9038949012756348  val_loss:0.28447607159614563  val_accuracy:0.8806629776954651  val_auc_1:0.8571106195449829 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 28.83it/s, epoch: 38/40 -  train_loss:0.21166759729385376  train_accuracy:0.9135237336158752  train_auc_1:0.922845184803009  val_loss:0.3054659366607666  val_accuracy:0.8872928023338318  val_auc_1:0.8537644147872925 ]\n",
      "100%|██████████| 29/29 [00:00<00:00, 29.38it/s, epoch: 39/40 -  train_loss:0.21334536373615265  train_accuracy:0.9162057638168335  train_auc_1:0.9161925315856934  val_loss:0.30080586671829224  val_accuracy:0.8872928023338318  val_auc_1:0.8560484647750854 ]\n",
      "100%|██████████| 29/29 [00:01<00:00, 27.09it/s, epoch: 40/40 -  train_loss:0.21065743267536163  train_accuracy:0.9156249761581421  train_auc_1:0.9142652750015259  val_loss:0.28072354197502136  val_accuracy:0.8872928023338318  val_auc_1:0.8580352067947388 ]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1.012537411848704, 0.7321080048878987]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "histories = []\n",
    "cost_time = []\n",
    "for sl_model in [sl_model_origin, sl_model_pipeline]:\n",
    "    begin = time.time()\n",
    "    history = sl_model.fit(\n",
    "        train_data,\n",
    "        train_label,\n",
    "        validation_data=(test_data, test_label),\n",
    "        epochs=40,\n",
    "        batch_size=128,\n",
    "        shuffle=True,\n",
    "        verbose=1,\n",
    "        validation_freq=1,\n",
    "    )\n",
    "    end = time.time()\n",
    "    cost_time.append((end - begin) / 60)\n",
    "    histories.append(history)\n",
    "\n",
    "print(cost_time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/OQEPoAAAACXBIWXMAAAsTAAALEwEAmpwYAABkC0lEQVR4nO3dd3hUVfrA8e87Jb0XAiSk0BHpoYMgoCBiw16xsqti/a11176uZd1V1+5a17LCgiIiiHREaugQSkJJSCCk90wy5fz+uJMQIAkhJCSE83meeTJz65sbuO+ccs8RpRSapmmadjxTcwegaZqmtUw6QWiapmk10glC0zRNq5FOEJqmaVqNdILQNE3TaqQThKZpmlYjnSC0FklEYkVEiYilHtveLiIrz0RcLYGIHBCRcc0dh9b66QShnTb3DatCRMKOW77JfZOPbabQqsfiJyLFIjL/DJ/3eRH5uoblSkQ6n8lYaiMiASLyloikuq/RXvfnsJPvrbVmOkFojWU/cGPlBxHpBfg0XzgnuBooBy4Skba1bVSfEsvZqqbfTUQ8gMVAT2ACEAAMBXKAQY1xDu3spROE1li+Am6r9nkK8J/qG4hIoIj8R0SyRCRFRP4iIib3OrOIvCEi2SKyD7i0hn0/FZHDIpIuIn8VEfMpxDcF+BDYCtxy3LEPiMgTIrIVKBERi4gMEZFVIpIvIltEZHS17e8QkZ0iUiQi+0TkD6cQxwlEZJmIvCQiv7uP+Wv1b+8icqv7euWIyJ+P29ckIk+6v/XniMgMEQlxr6usprtLRFKBJTWc/jYgGrhKKZWolHIppTKVUi8ppea5j3NMaUdEvhCRv7rfjxaRNPf1ywA+d1+bSdW2t7j/5v3dn2u9tlrLohOE1ljWAAEi0sN9474BOL5q5R0gEOgIjMK4Od3hXncPMAnoB8QD1xy37xeAA+js3uZi4O76BCYiMcBo4Bv367YaNrsRIykFARHAz8BfgRDgT8AsEQl3b5vpjjXAHf+blTe/03CT+1htAA/3ORGR84APgFuB9kAoEFVtvweAKzGuZ3sgD3jvuGOPAnoA42s47zjgF6VU8WnE3hbjOsUAU4H/Uq006T5vtlJqo4hEUve11VoQnSC0xlRZirgI2AmkV66oljSeUkoVKaUOAP/AuPEBXAe8pZQ6qJTKBV6ptm8EMBF4WClVopTKBN50H68+bgW2KqUSge+AniLS77ht/uU+dxlGCWOeUmqe+xv1QiDBHQNKqZ+VUnuVYTnwKzCynrHU5nOl1B73+WcAfd3LrwHmKqVWKKXKgWcAV7X9/gj8WSmV5l7/PHDNcVU9z7uvW1kN5w0FDp9m7C7gOaVUufsc3wKXi0hlFeNNGEkDTnJttZZF1xdqjekrYAUQx3HVS0AYYAVSqi1LASLd79sDB49bVynGve9hEalcZjpu+7rcBvwbQCmVLiLLMaqcNlXbpvqxYoBrReSyasuswFIAEbkEeA7o6o7DB9hWy7kd7n2riEjlZ3u1xRnV3pcCfu73x1wXpVSJiOQcF+sPIlI9aTgxSkE1/W7HywHa1bG+PrKUUrZqMSaLyE7gMhH5Cbgco9RXGW+t11ZrWXQJQms0SqkUjMbqicD3x63OxrghxlRbFs3RUsZhoMNx6yodxGhgDlNKBblfAUqpnieLSUSGAV2Ap0Qkw11PPhi46bhv2dWHNT4IfFXtXEFKKV+l1Ksi4gnMAt4AIpRSQcA8QKhZKhB73LI4jMSRfsLWJzrmuri/lYceF+slx8XqpZSqfuy6hmxeBIwXEd86tinl2A4Hxzfy13T8ymqmK4BEpVRytXhrvLZ1nF9rJjpBaI3tLmCMUqqk+kKllBOj6uRlEfF3tws8ytF2ihnAgyISJSLBwJPV9j2MUY3zD3eXTJOIdBKRUfWIZwqwEDgPo9qmL3A+4A1cUss+X2N8+x3vbjz3cjfGRmG0D3gCWYDDXZq4uI7z/wJ0dzc0W90NyH8DZimlHPWIfyYwSURGuHscvcix/28/xLimMQAiEi4iV9TjuJW+wrhpzxKR7u5rGyoiT4tIZbXPZoyEahaRCRhtGifzHcZ1uRejyqlSXddWa2F0gtAalbtuPqGW1Q8AJcA+YCXGjeMz97p/AwuALcBGTiyB3IZxc07EaIidyUmqRkTEC6Nt4x2lVEa1136MG+OUWn6HgxjffJ/GSAQHgccAk1KqCHgQI6HlYdSvz6ktBnd7ySXAHzAat7cD+Rg3zpNSSu0A7se4Vofd50yrtsnb7vP/KiJFGJ0FBtfn2O7jl2M0VO/CSKSFwDqMKsG17s0eAi5zx30zMLsexz0MrAaGAdOrLa/12tY3Zu3MET1hkKZpmlYTnbU1TdO0GukEoWmaptVIJwhN0zStRjpBaJqmaTVqNQ/KhYWFqdjY2OYOQ9M07ayyYcOGbKVUjUOdtJoEERsbS0JCbb0rNU3TtJqISEpt63QVk6ZpmlYjnSA0TdO0GukEoWmaptVIJwhN0zStRjpBaJqmaTXSCULTNE2rkU4QmqZpWo10gtA0TTsNLpdi5oY0Eg7kNncoja7VPCinaZp2ph3KL+PRGZtZs89IDhN6tuXJS7oTG1bXBH1nD12C0DStVcgpy+FMzm8zZ8shxr+1gm1pBbwyuRePXtSVFUlZXPTmcl78KZH80op6HWdL1hbuXHAn3+z8huKK4iaO+tS0mgmD4uPjlR5qQ9POPU6Xk3c2vcOn2z/lnl738GD/B5v0fAVldp77cTuzNx+if3QQb17fl5hQo8SQWWjjnwv3MCPhIP5eVh4Y05nbhsbiYan5u/jh4sPc8PMNlNpLsTlt+Fh8uKLzFdzY/UbiAuNOGotSin//to8im4P/u7hbg34fEdmglIqvcZ1OEJqmnQllFU7KHU6CfDwa7Zh5tjyeWPEEqw+vpmNgR/YV7OOl4S9xZecrG+0c1a3Zl8P/zdhCRqGNh8Z24b7RnbCYT7z578oo5OWfd/JbUjYxoT48MaE7QzuGEuhtxWQSAErtpUz5ZQppRWl8M/EbyhxlfLvrW+bvn4/dZWdY+2Hc3ONmRkSOwCQnnsPpUrz40w6+XJ3Cpb3b8a8b+mF2H/tU6AShaVqzcDhdrEzO5sfNh1iwIwO708WUobE8MLYLgd7W0zp2Yk4ijyx9hKyyLP48+M9c3vly7l90P+uPrOfjiz5mYNuBgPEt+9fEI3y3LhUFWM0mrGbBajZhMZnwsAgWkwkvq4kgHw8Cva0E+VgJ8vYgyMdKoLcVfy8LH63Yx4fL9xIT4sOb1/elX3TwSWNctjuTv83byZ4jRtWRxSSE+nkQ4utBccDn5Js2MCrgCQa3Hc7k/lH4elrIKcth5p6ZzNg9g8yyTDr4d+DyTpcTGxBLe7/2tPdrj685iIe+28yviUe4Z2QcT13SoyrxnCqdIDRNO2OUUmw+mM+Pmw8xd+shsosrCPCyMLFXO1xK8b8NaQT7ePDIRV25cWCHGr+Bn8zs5Nm8tPolQrxDeHP0m5wfdj4AhRWF3DbvNrLKsvh64tdYnBE8P2cHi3dlEhXsTYivB3anwu504XC6qt7bnS7K7E5sdled571xUAf+cul5+Hoe7d/jdDnZnbebSL9IAj0DT9jH4XSxdHcWqbml5BSXk11cztbimaTLbDwLr6DwyHAqHC7aBXrx50t7cGmvdogIdpedxamL+Xbnt2zK3HTMMUVZcVYEERcUxcAOnegR0oPrul13ytcRdILQtBZDKUV6fhntAr0bVB3QkhXZ7Hzy235+3JzOgZxSPCwmxnZvwxV9I7mwezieFjMAOw4V8OJPiazdn0vXCD+emXQeI7vUOB3BCexOO6+tf43pu6czuO1gXh/1OiFeIcdsk1aUxk0/34TT6UnOnj9gUn48Mq4rtw+PxXqSZGSzOykss5NfZie/1E5+aYX7fQU92gVUxWl32lmXsY7FqYtZkrqEHFsOQZ5BPDbwMS7reBkitf9tF6cs5uFlD3NZx8t4ecTLAGxMzePZH3ew41AhwzuH8sLlPencxr9qnxJ7CYeKD7H58D7+sWQthY4j9ItTuMx5HCo5RMfAjnw+4fN6XcPj6QShac3E5VLsyihi7f4c1u7LZd2BXHJLKhgcF8InU+Lx9zq9apaWIq+kgts+W8f2QwUM7RjKlX0jmdCrLQG1/H5KKRbsyODleTs5mFvGuB5teHpiDzqG+9V6jiMlR3h0+aNszdrKHT3v4MH+D2IxndhTf+2+HB6fO4cc/38RaOrIN5M+Izb0xG/2p6rMUcaq9FUsSl3E8oPLKbIX4W3x5oKoCxjWfhizkmaxNWsrQ9sN5Zmhz9DBv8MJx9idu5tb599Kl6AufDbhMzzNnlXrnC7Ft2tT+PuC3ZRWOLlzRBwPju2Cn7u0svlgPnd9sR6nUnw6JZ4BMUcTo91lx2pq2L8lnSA07Qw6mFvKgh0ZrNmXy/oDuRSU2QGICvZmcFwokUFevLdsLz3bB/DFHYMI8W2cRtsim52taQXszy6huNxBsc1BcbmDIpuDknL3+3IH5XYnSoFLKVxKVXtvHOeqfpE8OLbLCSWc7LJs3trwFt1CunFRzEW09W0LQFZRObd8spb9OSV8dMsALuzept4xlzucfP77Ad5dkozN7qRfdBB2p6Lc4aLc4aTc7jLem1JQEZ8ipnIiHbdzftAFRIf4EBPqQ4cQH6JDfBDglfm7mLkhjcggb64ckcl/9r7MpI6T+NuIv9X6rd7msLHm8BoScxKxOWyUOkqxOWyUOcqwOW1V75PykrA5bQR6BjI6ajTjYsYxpN0QvCxegFHVNGPPDN7e+DZOl5P7+97PLefdUpXEcm253Dj3RhwuB99N+o5wn5pLTTnF5bz+y26mJxykjb8nf760B74eFqb9dyPh/p58cccgOtWRSE+VThCadoYkZxZx9QerKSizExvqw+C4UAZ3DGFwx1Aig7yrtluUeIT7vt1ITIgPX989mIgAr1M6j9OlSMosYlNqPptT89l0MI+kzGKq/3c2Cfh5WvD3suLnacHPy4KvpwUviwkRMIlgEqn2HnJKKvgtKZvhnUN5+4Z+hPkZ33BzbbncteAu9hfsx6mcAPQO782QNhcyc0UwmXm+fDIlnuGdw06I1eawsa9gH3vz9+Jp9mRczLgTeuVkFZXzzpIkdmUU4WU142kx4WEx4Wkxkc9WNpb9Cy9TAL2tj5KXH0JqbilHCstPOJfFJEy9oCMPjOmCt4eZj7Z8xLub3+X+vvfzxz5/rNqusKKQFWkrWJK6hJXpKylzlAHgafbE2+KNl8XL+Gk2fnpbvOng34FxMeMYEDGgxpJLpYySDF5e+zLLDi6jR0gPnh/2PF2CunDPwnvYnr2dLyZ8UdVmUpdN7mqnbekFAPSKDOSz2wcS7u95kj1PTbMlCBGZALwNmIFPlFKvHrc+BvgMCAdygVuUUmnudVOAv7g3/atS6su6zqUThNbcMotsXPXeKsodTr6bOuSYOuSarN6bw91frifEz4Nv7hpCdKhPnds7nC7+sfJHpu9/m/KMKynJ7wJAoLeVftFB9O0QRL/oYLpF+BPgbcHbaq6zLrw2MxIO8szs7QT5WHn3pv50bWfmrgV3caDwAO+PfZ8I3wgWpizk572/kFywG4BY/25c2eUS4tvGc7j4MEn5SSTnJbO3YC8Hiw7iUkcbfwe3G8yLw16kvV/7k8eyewYvr32ZbsHdeH/c+4R5H01ANruTtLxSUnNLSc0pJbu4giv6tqdLxNHrrpTiL7//hTl75/DMkGcAWJy6mHWH1+FQDsK8wxjTYQxjo8cysO1ArObGqfJTSrEwZSGvrHuFPFse3UO6syNnB6+OfJVLO15a7+M4XYrp6w+yO6OQxyd0P6ZxvLE0S4IQETOwB7gISAPWAzcqpRKrbfM/YK5S6ksRGQPcoZS6VURCgAQgHlDABmCAUiqvtvPpBNG6rN2XQ16pnQnntz2j580tqWDJrkyOFNqYMiy2qv73ZIrLHdzw8Wr2ZZXw3dQh9I4Kqtd+mw/mc/vn6/Awm/j67sF0jTgxqVQ4XPywKY33l+0ly+8NzD6pgDC+7d3c1/9O4sJ8G5QI6pJ4qJD7vtnAwYJcYnt+TZ4jhXfHvMuwyGEA7Msq5uZP1lLiyuT6UXnsKFjJtuxtVfubxERMQAydgzoffQV3ZsORDbyx/g1EhMfiH2Nyl8k1xu5SLt7e+Dafbf+MC6Iu4O8X/B0fa90JtDYVzgqmLpzKhiMbAIj2j2Zs9FjGRI+hd3jvGp8xaCwF5QW8ueFNZiXN4u5ed/NQ/4ea7FwN1VwJYijwvFJqvPvzUwBKqVeqbbMDmKCUOijGv5ICpVSAiNwIjFZK/cG93UfAMqXUf2s7n04QrUNBqZ2X5yUyIyENgL9f05tr409s7GtM+7KKWbTzCAsTj7AhJa+qLr5zGz8+unXASet77U4Xd3+ZwMrkbD65Lf6U6uABdmcUceuna6lwuvjyjkH06RAEGPXzMxLS+HDZXtLzy+jcIY8jfq/xYN8H2Zm3k4UpC7my85U8M+QZPMyN9/BZpcNF+Vz9/e0UqgP0kGl8ct0UAr2t7M4o4uZP1qKU4qu7BnNe+wAADhUfYkfODqL9o4kNjD2mAba69OJ0nv39WdZlrGN4++E8P+z5qvYMMG7of1n5F+YfmM91Xa/jqcFP1VmlUx8F5QXM3z+fARED6BzUudET6snklOUQ4hVyxs9bH82VIK7BuPnf7f58KzBYKTWt2jbfAmuVUm+LyGRgFhAG3AF4KaX+6t7uGaBMKfXGceeYCkwFiI6OHpCSktIkv4t2Zszfdphn5+wgt6SCe0Z2ZHt6Aav35TToplsTp0tRZLNTUGbnUL6NZXsyWZR4hL1ZJQCc1y6AC3uEYPddxuas9SQlXoq9PIB/XNeH8T1rLskopXhi1lZmJKTx6uRe3DAoukGxpeSUcMuna8krsfPuTf3Yn13CR8v3kVFoo2+HIB4c25lFWW+zOHUxi69djI/Vhw+2fMCHWz6kX5t+vDn6TUK9Q+s8R74tn/kH5mMWM+Njx9fYZ79Sqb2Uexfdy5asLUxs8zgzVgTRPsibh8d14aW5iVjNJr69Z/BJq9Fq41Iupu+ezpsb3sQiFp4Y9ASXd7qcwopCHlzyIBszN/LIgEe4o+cdLfKm2pq05ATRHngXiANWAFcD5wN3U48EUZ0uQbQsBaV23l+eTLcIf0Z0DqNNHY2wmYU2nvlxOwt2HKFn+wBeu7o350cGVlXb7M0s4dt7BtfryVWA/NIK/rlwD/uySigos5NfVkFBqZ2icscxjbgWkzC0UyjjekQwtkcbUks388q6VzhQeACLyUI7nyhMGfez/aCTaRd25pGLup7Qs+etRXt4a1ESD47twqMXdW3QtaqUUWDj1k/XkpRpPHU7KDaEB8Z2ZkTnMHJtuVw08yKu6XoNTw9+umqfXw78wjMrnyHYK5h3xrxDt5Bjx+NxKRdrDq3h++TvWZK6BLvL6FFlNVkZFTWKyztdzoioEcd0kbQ5bExbPI31R9bz2sjXmBA3gQ0peUz7diOHC2xEBnnzzd2DG2XE0oOFB/nL739hY+ZGRkWNIrUolbSiNF4e8TKXxF1y2sfXTq7FVjEdt70fsEspFaWrmM5uLpfi7v8ksGRXZtWy7m39GdkljJFdwhkUF4KX1YxSRgPcy/N2UuFw8fC4rtw9Mu6Yh5myisq5+oNVFNnszLx32Emrezam5vHAt5vILLJxfmQgQd5W99AJHgS43wd6Wwn19WBAbDABXlbSi9P5+/q/szh1MTEBMTw56Ek8zZ7cu+heOgZ2Irr8UWYmZHNB13D+dUPfqrGEZiQc5PGZW7lmQBR/v6Y3IkJBeQHPr3oeEWFav2l0DOx4Stcut6SCD5fvZUz3NgzpeLREUNkbZ86Vc04YxG1Hzg4eXPIgRRVFvDLyFcZGj+Vw8WFmJ89mdvJsDpUcItAzkEkdJ3FV56twKRdz9s5h3v555NpyCfEKYWLcRC7vdDmdgjrx4NIHWZW+ipdHvMxlnS47Jrb/rD7AtfEdjumRdbpcysU3O7/h7Y1v42n25O0L3ya+bY33K60JNFeCsGA0Uo8F0jEaqW9SSu2otk0YkKuUconIy4BTKfWsu5F6A9DfvelGjEbqWmfk0Ami5Xh/WTKv/7Kb5y47j0FxIfyWlM1vSVms359HhdOFh8XEoNgQKpwu1u3PZVBcCK9O7lXrQ1IHsku4+oNVeFnNfH/fsBq7hLpcik9W7uP1X3bTNtCL927qX1WXX5tyZzmfb/+cT7Z9gklMTO09ldvOu62qPn/5weU8tPQhBkQMYHTgU7z0UxIRgZ58eMsAsosruPOL9QzrFMpntw/Eajaxv2A/Dyx5gPTidDzNntgcNiZ3mcx9fe87pvfNqbK77EyYOYEuwV348KIPa9wmqzSLh5Y+xLbsbfQO7822rG0oFEPaDWFyl8mMiR5zQpuA3WXn9/TfmbN3DssOLsPushPkGUR+eT4vDHuByV0mNzjmhsgoyUAQInwjzuh5z3XN2c11IvAWRjfXz5RSL4vIi0CCUmqOuxrqFYyeSiuA+5VS5e597wQqy9IvK6XqfI5cJ4iWYfXeHG7+ZA0Te7XjnRv7HVN/XFbhZO3+HFYmZfNbUjZZxeX838VduXFg9EkHGtuals8NH68hOsSHGX8ceswTunklFfzf/7awZFcmE3q25bVretc5EJxLuVh+cDmvr3+dtOI0xseO50/xfzqmobTS3H1zeeq3p7iww4Xc1vEZpn27lbzSCswmITbUl+l/GIK/l5Xf03/nseWPYTVbeXP0m8QGxvLRlo+YsXsGVrOV23vezu09b29QT5xf9v/CYyse472x73FB1AW1bmdz2Hhl3StsPLKRCXETuLLzlUT6RdbrHAXlBfyy/xcWpCzg0rhLubrr1accp3Z20g/KaWdEZqGNif9aSYC3hTnTRtS7i2h9rdiTxZ1frCc+Npgv7hiEl9VMwoFcHvjvJnKKK/jzpT24bWhMjY2adpedDUc2sChlEUtSl5BVlkWnwE48NfgpBrcbXOd5v9n5Da+ue5UrOl3Bg33+zCPTt5CWV8aMPwyljb8nX+/8mjcS3qBzUGfeGfPOMf37UwtTeWvjWyxMWUioVyj39b2PyV0mn1KvnFvn3UqOLYe5V81t0i6Z2rmprgShpxzVGoXD6WLafzdRUu7g23sGN3pyALigazhvXNuHh6dv5tEZmzk/MpB//LqHqGBvZt07jF5Rx/bKKXeWs/rQahalLGJZ2jIKygvwtngzInIE46LHcVHsRfUav+bmHjdTWFHI+5vfx9/Dn6/vehyXAqey89yq5/gh+QfGdBjDKyNfOaGEEB0QzT9H/5MtWVv4Z8I/eWnNS3yV+BV/G/E3eoX3Oum5E3MS2Zy1mccHPq6Tg3bG6QSh1arQZsdqMuHtYT7ptv9YuId1+3P553V9anzYq7Fc2S+SrKJyXlm8hF9T0+ndw5uLzg9medYOfjlcRqmjlDJHGQXlBWw8spFSRyn+Vn9GdxjN2JixDGs/DG/LqTew/rH3HyksL+TrnV8T5BnENV2v4dFlj7IxcyNTe0/l/r7313kD7xPehy8mfMHSg0t5bd1r3Lv4Xr665KuTzhr27c5v8bZ4N9kEOJpWF50gtBql55dx5Xu/Y3e6uHN4HFOGxdZar78o8QgfLNvLjYOimdw/qknjsrvs2Px/xq/jJyhcJLsgeauxrnLMHB+LDz5WHyZ2nMi46HEMajvotIdQEBEeG/gYBeUFvLv5Xb7Z+Q2ljlJev+D1enfHFBHGRI+hS1AXbpl/C/cuupevJ35dawN2ri2X+fvnc1WXq/D3aLqkq2m10QlCA4yRKMud5dicNnJKivnjN2uxSRHdoz341+rN/HtjBfGdvOkZZcWuSiiqKKLIXoS/OZyf1gtdojrx9MQuTRrjgYIDPPXbU2zP2c4Vna5gau+p+Fp9qwZXa+oqGJOYeGH4C9icNrZlb+ODcR/QM6znKR+nQ0AH3h3zLncuuJNpi6fx2fjPamy8nrVnFhWuCm7qflNjhK9pp0w3Up/D3trwFjP2zMDmsFU9QFUfVvEh1DsIX6sP+/IPosQYVdNqstI5qDM9QnvQPaQ7nYM6Y3PYyC/PJ8+WZ/wszyPflk9+eT4AF8dezMS4iXU+1auUYmbSTP6+/u9YTVaeG/ocF8defFq/++lQSuFSLsymk1e91WVp6lIeXvYwIyNH8taFbx3TcG132ZkwawKdAjvx8cUfn27ImlYr3YvpHFRW4eSLVQfYlJrH0xN7nPDU6+zk2Tzz+zOMihpFp6BOeJm98LR4siQxl7V7i7k+viMXdo3Ex+pDgGcAAR4B5BSa+XJlBnO2ZGA2Cd0i/NmWnsfL17YjJCSLnbk72ZWzi525O6sSQHVmMRPkGWS8vIIorCgkKS8JD5MHY6LHcGXnKxnSbsgxN95cWy7PrXqOZQeXMaTdEP46/K+tqp/8d7u+4+W1L3Nt12t5ZsgzVT2wFhxYwJ+W/4l3x7zLqA6jmjlKrTXTvZjOIQ6ni/9tSOOtRXs4UliOp8XEqr05/G1yLy7vY3S/3Jmzk7+u+SuD2w4+5pvrF7/vZ+XGRKZe0JGnL+pxwrE7+EPf69vzyLhSPli+l5kbDnLf6C7cPKA7QFVdvFKKI6VH2FewD1+rL8GewQR5BeFn9TuhGmhnzk5mJ8/m5/0/88uBX4jwieDyTpdzZecrSSlM4Znfn6GooojHBz7OzT1ubnU9eW7ofgOHSg7x+fbPae/Xnrt73Q0YjdNRflGMiBzRzBFq5zJdgmgllFLM357BGwt2sy+7hP7RQTwxoTtRIT48+N9NbEjJ48ZBHXjooihu//UmHC4H0ydNrxrgbcmuI9z9ZQLjekTwwS0D6jVfss3uxMt6etUslSqcFSw9uJTZybNZdWhV1fwBXYK78OrIV+kafHrjHLVkLuXiyd+eZP7++bwy8hU6BXbiurnX8Vj8Y9zW87bmDk9r5XQVUyu3Kjmb137ZxZa0Arq08ePxCd0Z16NNVXWF3enizYV7+GB5EqEdv8bpuYcvL/mS3uG9AWPs/2s/XEVcuC8z/jAUH4/mLVgeKTnC3H1zAbjlvFtqHTa6NalwVvDHRX9kU+Ymzgs9j6S8JBZdu4gAj4DmDk1r5XQVUyuVV1LBQ9M3s2JPFu0Dvfj7Nb2Z3D/qhG//VrOJxyd0J8c6l3lpiTiPXMWuA8H0ClNkFpVz15fr8fey8umUgc2eHAAifCO4q9ddzR3GGeVh9uCtC9/itnm3sTVrK9d3u14nB63ZNf/dQGuQynkI1uzN4S+X9uCWITF1VvesSFvB/LT/cHH0pRy2X8Ljs7by+97sqiGx//fHoac8L7LWuAI8Avhg3Ae8u/ld7jr/3EqQLVbOXtj+PSTOBr8IuOJdCDj5VKmtha5iOktNX5/KE7O28eeJPbjngrqHlD5YdJDr515PpF8kX13yFVaTJx8sS+afC/cA8MmUeMZ0bz09g7RWymkHlxNMZhAzmJqow0JBmpEUts+Cw5uNZR0GQ8Z28PCBqz+Fjq2nZ5muYmplDmSX8MJPiQztGMpdI+oeqsHmsPHoskcB+Ofof+JlMUoJ08Z0YUSXcArL7FzQNbzJY9bqoBQUZUDmDjiSCLn7wCvQ+Kbq3+7oT78IMDfDf1mXCzK2QPIiOPA7VBS7b9YO9087OB3GT5MVIvtDzHCIGQptep78Rl6aCxnbjFdBGtjyoSwfbAXV3ueDvfTEfcUMJjPOCguFaZ5YfCx4hnthDfFCPH3A6gNWb7B4Gz89/cDDDzx8j/3p6QcF6bDje0hdbRy7fX+4+GXoeRUERkLWbph+K3x1JVz4ZxjxaNMlqeoyd0JmIoR2hrCuxu9xhugEcZZxOF08PH0zFpPwj+v61DlMtlKKl9a8xK7cXbw39j06+B87t3Pfk8yXoAF2GxSmQ+Eh989040ZitkLHCyF2hHFzqS9HORzeYtwMMxON//xHdhg3wErewVBeZNyAqxMT+LYB3zAjqbgc1V7Oo+9RYLIYN2uT2Yi1+mffMAjrBuHuV1hX8Ak59lylubB3iZEUkhdDiXvyp7a9wCfMfUyrkbAqj222QEUpHFxrVMmAkeg6DIGYYcbLv63x+x7eChlbjZ8FqUfP6xkAXkHGft5BENLR+Fm5zGQ2EpZygsuJo7CE3MU7yVuRjKu88no5EUspHqFOPMPK8Awx4RnkwtO/AqtnKeIoAXtJzX+fNufBmL/A+VdDSEecxcXYEhOx7ViIOcCfgJt/xrT4SVjyEhxcB1d9eOK1awwlObB9Jmz+9mgpBgCB4FgI7w5tuhs/w7sZf0+PUx9K/mR0FdNZpnKKy3du7MdlfWquC7U5bCxPW86cvXNYkbaCe/vcy3197zvDkZ6eoiVLKV2/Hmu7dlijIrFGGi+z3yncjGtTlmd8Sy/NNV5lucay6u9Lso1kUJpz4v7eIeCwGd9ozR4QPRQ6j8Pu34eC3xMp+PlnxGzBf/zFBIweiqc5A1LXGDfO9I3gNJ48xzMA2vQwbkptzoOI8yC8B/iGGjfB0mwoOgyFh6Ho0NGfpblGsjBZjnuZjZ9QLWlU/6bv/lx4GHKSjN+hkm+4cbMJ6QhZuyBtPSiXkaw6jYHOF0HnseBXj7nBlYL8VOObeMrvkLLaON8xBEI7Qdve0K63+2cfI3nVgz0zk9zPPidv+nSUzYb/hPGE3nknAOVJyZQnJ1OelER5cjKOw4ePntVqxRoTjWdsHB7RkXhEtcUjMgyPiCBMvkHYMsuxbd9O2Y4d2LbvoGL/fqrPU2vy9yfwyisJPt+M55a/GyW76740Sk21qfz35uFnlAK9AqGmebYdFZC80EgKexYYf6u2vaHvTca/sbz9kLnL+Ptk7cZ1JBl7kcJeYobgWPxeW1eva3c83c21ldiUmsc1H67m8j7tefP6vsesc7qcJBxJYO6+uSxKWUSxvZhw7/CqGc3OlgfMXDYbR159lfzvpoPFAo5jv0WbAgOxRrbHIzIS3+EjCLrmasRSQ0HYXmb8p8xJdr/2Hn1fmoPLCcohuFwCLsHlFJRHAMoSgLIGIN6BWKOjsbSPg4BI4xUYZdwQPHyMkkDqGlw7fqFo4a/kb8mj9IgHIPjEBaEUlB3IB8AzyE5ATAUB8R3x6D0coodAu77G8Wq6UTQBV1kZ5cl7KU9KQlWUYw4MxOxhx6JyMdszMJelILnJxo08ONZICF0ugsgBRuI5XcVZRsIoyYSI842Xu+TlKi/HfvAgFampqAo7lvAwLKGhmMPCMPn6HjO/h/3QIXI++ZT8mTNRTieBky4ldOpUPDt1qvXUzuJiKpKTKd+7l4r9+ynff4CK/fupOHgQ7DUPMWOJiMCrZ0+8zu+J9/nn49WzJxUpKeR98y2Fv/4Kdjs+/XoSHLYT/7BMZOKr0OOyqpu3M2U7th3bKNubju1wOeUFFqx+TrxD7Hi3Aa+YICxh7cA/AvzaGiWixB+NLyS+baD3dahe1+MwtaHiQAoVqSnY09Kxp6djT0ujIj0dZ3Z2VbxeHSOJm7eoQX+a5pxRbgLwNsaMcp8opV49bn008CUQ5N7mSaXUPBGJBXYCu92brlFK/bGuc7X2BFFS7uDSf/2G3amY//DIqhnVdufuZu6+uczbN4/Mskx8rb6Mix7HpE6TGBgx8LTHC6o1nrXryHztNazR0YTcegve/fufOFGP3cbR6g7LSW+G5cnJpD/yKOVJSYTefRfhDz6Is6QEe1qa8R/D/apIT6di/37sqQfxiIqgzXUj8OtoRfJTjW+u+SlQfOTYg/u3g9DOlDsiyPk9m4K1yca39JMwBwXhERdnvGJj8YiLxTMuDmdeHvmzZ1M0/xdcpaVY27clcGhnAjsU4JG3BgTsgf0pygimcFsWZYl7AfA8rwcB4yfg1aM75pBQLCHBmENDMXk2zrMeyuk0boJ79mBLSjK+Re9Jwn7w4DHfhE8ggjkgAHNwMNb27bHGROMRHYNHTDQe0dFYO3Q4IUalFK7CQhw5uThzc4yfhQWIyWQkbYsFsVgRqwWxGC+XzVZ1w6tIMV6Owxm1xiaenlXJwuznR8n69QAEXXkFoffcg0d0dMOvlcOBPS2N8v37qdh/AFdZKV7nnYdXz55Y29ReUnJkZ5M/cxZ506fjOHwYi7+FoJhcTBaFLdeDslwr9uKjX1qsbQLwjO2A/XAW5WmZxvyZgDXYincb8A4qwdPPhj2gDxXmTlQUChUpqUbCLCs7emKL5ZgStUdUFNbIKON9hygs4Q1rS2yuOanNGHNSXwSkYcxJfaNSKrHaNh8Dm5RSH4jIecA8pVSsO0HMVUqdX9/ztfYE8dT3W/lu/UG+u2cIgzuG4nQ5eXvj23y+43MsYmFE1Agu7Xgpo6NGVzVENwVlt5P17nvkfPwx1vbtcRYV4SosxOu88wi+9VYCBnfDtP9X2PWzUUdLtX9fVfXgFqO+2uwJXoEoT38KdisyFhzB5GGh/ZQh+PXvDojxjao0x6jyKc12v89BlRdRfMiTzM0BVBRZ8QmvoM1Ib7y7xUFQjPEK7Wg07IV0pCwplZyPPqJo0SLEy4ugyZPxiIlGPDwQq/WEn8ruMG5g+/cbN9wD+3FmZR9zLUw+PvhPmEDQVVfiPWAAUtlgWZl4qjVg2g8fpvCXBRT+Mh/blq0nXFeTry/mkBAsISGYw8PwiY/H74JReMTF1jhD3jF/E5eLss2bKfx5HoW/Ljgap9mMR0wMnl274tmlM55duuDVtSvi44MzL8945ebiyMvDmWt8duTmGkk4JQVXYeHRk4hgadsWa/v2uEpLcebk4MjLq/Ub+MmYAgPxiIk57mX8PRzZOThzsnFk5+DIzsaRk40z2zifT79+hN59F9b2zd/VVDkcFC9fTt6331Ly+yoArG1CjJJHv/iqkoc58OhAlM7iEqMKa+tWyrZuwbZlK46srKMHtVjwiIoyrof7C4lHTAwe0dFYIiJqLi2fpuZKEEOB55VS492fnwJQSr1SbZuPgH1Kqdfc2/9DKTVMJ4hj/bojg6lfbeCPozrx5CXdKbGX8OSKJ1mWtoxru17LA/0eINgr+NQOmpZA/ttPUnYgk+DxQ/EacYXR4OpV+8NZFQcPcuhPj1G2ZQuBV0+m7dPGlOEFX71H7nezqMgoxOzpJLhzKUHDYrD2nWD0EjmhLtzdmOqw4czPI2P2HgoTi/CJNNF+pA2rqcDYHsDiZTSK+oaCT6jx3ifU+OzfDuUXSf6KRLI+/RZnbi4Bl15K+COP4BFlzMVcun492R99TMnKlZj8/Qm+5WZCbrsNS/ApXi/AWVRkfPs9sB9MJvwvvBCTz6k3DNqPZGI/lG7cnHNzcebk4sjNMW7SuTnY0w9RkZICgLVDB/wuuAC/URfgM2gQJi8j+SulsG3fTuG8+RTOn48jIwPx8MBv1Cj8xozBq3s3PDp2PK2SiTM/n4rUVPe32RTsqanY0w8ZySwsFEtIKObQEOMbfoj7Z0AAyqXAYUc5HMbL7kA57OBwIJ6eeERHYw4KanBcLZH98GHEwwNLaOgp7aeUwpGRQUVKilE6aN8esZ7e3CWnqrkSxDXABKXU3e7PtwKDlVLTqm3TDvgVCAZ8gXFKqQ3uBLEDowRSCPxFKfVbDeeYCkwFiI6OHpDi/k/VmmQW2Zjw1m+0DfBi9v3DybZlMG3JNPbl7+OJQU9wY/cbT+2AaRtg2Svkzf+djA1BVYt9IsoJ6V6G36DeSOexRg+d9v2Mb/pKUTDrOzL+9ncQaHfXeAJ6BhtVOclLoOgQCjOl9CV3lzfFW/aD2Yz/uHF4du2CJTTsaL1yaBiWsFBMXl6UbdtO+v/9H/b0dMIfmEboPfcgZrNR3eCwGYnFw7de9fTO4mJyPvmE3C++BKeToGuvxbZ7N2UbNmAOCSHk9tsJvunGxmnkPgMq0tIp+W0FxctXULJmDcpmQzw98Rk8CM+4OIqWLDWqjaxW/IYPJ2DiJfiNGXPW/H5ay9GSE8Sj7hj+4S5BfAqcD1gBP6VUjogMAGYDPZVShcefp1JrLEEopbjzi/Ws2pvD3AdGUCJ7eWjpQ9iddt4Y9QbxKppDTzxJ6F134T/mwroPlrYBlr8KSb+SlxpOxiorfqNG0u6vL5P//Szy/vMljpx8PILMBHfKJSi2FJN/AE4J4MiyEgr2e+EdVkH7IXl4+DmNY/qEGr0ruk+CruOruvtVpKaS9823FMydizOnhl5AgMnPD5fNhiU8nMg3/o7PgAGNcs3sGRlk/esdCn74AUvbtoTedRdBV0/G5H3m+o43Nld5OaXr1lO8YgXFK5ZjT0vHd8gQAiZegv+4ccdUYWjaqWrJVUw7MJLIQffnfcAQpVTmccdaBvxJKVVrBmiNCWJ7egGT3lnJU5d0p13UDp5f9TztfNvxzth36BjYkcy33iLnw48ACLvvXsLuv9/4Bl5d+gZY9hokLQDvYPLVeA5/+Ru+I0cQ9e67mDw8AKNtoWjhQnK+/BLblq2YfLwI7B9O8c4s7Lk2wi4fRNjNlyNBUUZfdv+2YDl59YWrosKor87Ocdclu+uWc3IQq5Wwqfc0SXWDIy8Ps68v4v79WhNVUdEqfy+teTTXk9TrgS4iEgekAzcAx8+dmAqMBb4QkR6AF5AlIuFArlLKKSIdgS7AviaMtUX6aeshLCZFntds3l35BYPaDuKfo/9JoGcgSikK583HZ+BArFFRZL//AWXbthP512cw52+Hfcth3zLI2mn0ZR/zDAXZMRz+83P4Dh1K1DvvVCUHMPqHB0ycSMDEiZRt3kzuf/5D3oJfsbRpQ8xXr+MTX+O/n5MyeXhgatcOa7t2jXRV6qchbQxnC50ctDOlyRKEUsohItOABRhdWD9TSu0QkReBBKXUHOD/gH+LyCMY3V1uV0opEbkAeFFE7IAL+KNSKrepYm2JlFLM3XqIdl1n8s2uDVzT9RqeHvw0VpPRgGXbkYg9NZWwu+8kcGAk3p7pZMz4jf0TxxA1IhevcItR/dP/Vuh3K4VLVnLoL4/hM2gQUe+9W2fjpXffvkT27Uvb/HzE27vRumBqmnZ2adKhNpRS84B5xy17ttr7RGB4DfvNAmY1ZWwt3eaD+RwuPYCvbOCeXvfwQL8HjunuWDhvHljM+Cf+CUksJFhMeF3fk7T5pRxY6k27558lcPI1xrYLfiX9scfx7t+PDh+8X+/6+NbW00TTtFOjx2JqoeZuPYyH334Aru569THJQblcFM75Hr82pZjbdYQLHoPYEXh7BxF3fzbpjzzKoaefoWz7TnwGDSL9T3/Cu3dvOnz4UYO6ZWqadm7SCaIFcrkUP289TERkOp6+7Yj0izxmfdmcD3Bk5xMwoQ3c9qMxmJmbJSyM6M8/I/ONf5D7xRfkffstXr170+HjjzD7+Z7h30TTtLOZThAtUEJKHhmFZbSNSWJ4xAXHrtw5l8LPXkcsPvj95ftjkkMlsViIePIJvPv2oXjpUiKefhqzv/+ZCV7TtFZDJ4gWaO7WQ3j5ZFPiKCC+bbXeQzt/Qk2/ncL0tviNHoU5rO7hBgImTCBgwoQmjlbTtNbq7Bji8xzicLqYt+0w3eOM8VniI9wJIvFH+N/tlLrOw1niImDS5c0YpaZp5wKdIFqYtftzyS6uwC8whTbebYxJfnbMhv/dAZEDKHSOQHx88BvVeqY81DStZdIJooWZu/UQvh4m0sp2MKDtACRxNsy8E6IGoq6fTtHiZcYgcWfx0BGapp0ddIJoQexOF/O3ZzCsuyLHlk28ZwTMuhs6DIJbZlKyaTvO/HwCLp3Y3KFqmnYO0AmiBfk9OZv8UjsdIjMAiN/wX2N2qRv/C57+FM6bj8nfH98RI5o5Uk3TzgW6F1ML8tOWw/h7WShmN6EmT+Iyk+CW78E7GFdFBUWLFuE/btwxYyhpmqY1FV2CaCHKHU5+Tczg4vMi2Hh4NQMK85D4u4yJ4oGSlStxFRURMPGSZo5U07RzhU4QLcSKPdkU2RyM6FTKkfI84k2+cPFLVesLf56HOSgI3yFDmjFKTdPOJfVKECIyQkTucL8Pdw/hrTWiuVsPEexjRe39JwDxo543ZlMDXGVlFC1div/48Wd8OkJN085dJ00QIvIc8ATwlHuRFfi6KYM615RVOFmUeISHovezIXMDQSYPOvW4smp98fLlqNJSAi7R1Uuapp059SlBXAVcDpQAKKUOAXpgn0a0dHcm1op8bsr4Oxt8/RkQORyTHP3TFP48D3N4GD4DGzZpj6ZpWkPUJ0FUKGNeUgUgInpI0EY2d+shXvf+kmxnIekmRXzbQVXrnMXFFK9YQcD4CSdOJ6ppmtaE6pMgZojIR0CQiNwDLAL+XZ+Di8gEEdktIski8mQN66NFZKmIbBKRrSIysdq6p9z77RaR8fX9hc42JeUOvHbP5mK1ig19rwY4ZoC+4iVLUOXlBEzUD8dpmnZmnfQ5CKXUGyJyEVAIdAOeVUotPNl+ImIG3gMuAtKA9SIyxz2LXKW/ADOUUh+IyHkYs8/Fut/fAPQE2gOLRKSrUsp5ir9fi7dy4zaelU8pDu9HQmAY/oX+dAnqUrW+cN58LO3a4d23TzNGqWnauag+jdSPAolKqceUUn+qT3JwGwQkK6X2KaUqgO+AK47bRgEB7veBwCH3+yuA75RS5Uqp/UCy+3itTpsVT+Mldnyu+zcJmRsZ0GYAZpNRlWQ/dIji338nYMIExKR7JGuadmbV567jD/wqIr+JyDQRiajnsSOBg9U+p7mXVfc8cIuIpGGUHh44hX0RkakikiAiCVlZWfUMq+VYumY9/cpWsanDFLJ9A0ktSj2meinzjX8gJhMht9zcjFFqmnauOmmCUEq9oJTqCdwPtAOWi8iiRjr/jcAXSqkoYCLwlYjU+6uyUupjpVS8Uio+PDy8kUI6Mw5kl7Bz/oe4EAZc+QAJGQnA0fkfShMSKJw3j9C77sIaeUJu1DRNa3KnUm+RCWQAOUCbemyfDnSo9jnKvay6u4AZAEqp1YAXEFbPfc9aNruT+79O4ApZTkWHkXiExpBwJAFfqy/dQrqhnE4y/vY3LG3bEnrP3c0drqZp56j6tEHcJyLLgMVAKHCPUqp3PY69HugiInEi4oHR6DznuG1SgbHu8/TASBBZ7u1uEBFP91PbXYB19fuVWr6X5ibin7mOSLLwGngbAAlHEujfpj8Wk4X877+nPHEnbR77k573QdO0ZlOf0VyjgIeVUptP5cBKKYeITAMWAGbgM6XUDhF5EUhQSs0B/g/4t4g8gtFgfbv7mYsdIjIDSAQcwP2tpQfTj5vT+WZtKj9FboTiAOh+Kdll2ewv2M+Vna/EWVhI1ptv4T1ggO7aqmlas6ozQbi7qk5WSj3dkIMrpeZhND5XX/ZstfeJwPBa9n0ZeLkh522p9mYV8/T32xjRwZPz85dBr2vBw4cNB34DjPaH7Pc/wJmXR8S/P0ZEmjdgTdPOaXVWMbm/te8WkegzFE+rVVbh5P5vNuJpNfNuv1TEXgr9bgEgISMBb4s3nQq8yP36a4KuuRrvnj2bOWJN08519aliCsao8lmHezwmAKXU5U0WVSv03Jzt7Moo4os7BhL0+z8htDNEDQSM9od+bfqR8/obmLy8CH/44eYNVtM0jfoliGeaPIpWbtaGNGYkpHH/hZ0YHVYEqatg7LMgQp4tj+T8ZG7O607Jitm0eeIJLKGhzR2ypmlavYbaWH4mAmmtko4U8ZfZ2xkcF8Ij47rC8ldATNDnRgA2HtmI2ano8d+1eMTFEXLzTc0csaZpmuGkCUJEinCP5Ap4YMwHUaKUCqh9L63Sa7/sxtvDzDs39sMiwJb/QscLIaA9YFQvXbbRjOngYSI+/gjR801rmtZC1OdJan+lVIA7IXgDVwPvN3lkrYDD6WLNvhwmnN+WNgFecGAFFByEvkdLCYlJq7l6pRPfURfgd8EFzRitpmnasU5pBDhlmA202uG3G9P2Q4UUlzsY2tHdprDpG/AMhO6TAMguy2bgnD1Y7YqIJ08YDV3TNK1Z1aeKaXK1jyYgHrA1WUStyKq92QAM6RgKtgLY+RP0vRGsXgCsTlzAqG0K0+RL8IzT03xrmtay1KcX02XV3juAA5w4bLdWg9V7c+ga4Ue4vyds+A4cZdD3lqr12T9+T2cXxN36h2aMUtM0rWb16cV0x5kIpLWpcLhIOJDH9QPdYw5u/gbCukFkfwDsTjuRy3eTExdMj25dmzFSTdO0mtXaBiEifxeRE77aisgfROTVpg3r7LclLZ8yu9OoXspOhoNrjcZp9/AZO1b9RIcjTuTSsc0cqaZpWs3qaqQeA3xcw/J/A5OaJpzWY1VyDiIwpGMIbPnW/ezDDVXrj/zvv1SYoecNunpJ07SWqa4E4ekeWfUYSikXoEeRO4nV+7I5r10AQV5m2Pxf6DwO/NsC4CovJ+y3nST1CSEoLKqZI9U0TatZXQmiTES6HL/Qvays6UI6+9nsTjam5DOsUyjsWwZFh4559uHQLz/iU+ZETbyw+YLUNE07iboaqZ8F5ovIX4EN7mXxwFPAw00c11ltY0oeFU4XQzuFwuY3wCsIuh2d2yFj+teUBcD5429sviA1TdNOotYShFJqPnAlcCHwhfs1GrjaPc+DVotVe3Mwm4SBEQI750Lv68DiCYA9IwOfTUms6+dH97DzmjlSTdO02tXZzVUptR2Y0tCDi8gE4G2MGeU+UUq9etz6NzESEIAP0EYpFeRe5wS2udelnk3Di6/el0OvyED8k34EZ3nVvA8AeT/8gCiwXzJCTwikaVqLVp8H5RrEPRvde8BFQBqwXkTmuGeRA0Ap9Ui17R8A+lU7RJlSqm9TxddUSsodbDmYz9QLOsLG/0Db3tCuDwBKKbJmTmdnNPTrO6GZI9U0TavbKY3FdIoGAclKqX1KqQrgO+p+AvtG4L9NGM8Zsf5ALg6XYlxwBmRshX63Vq0r27ABU/oRlvexMLT90GaMUtM07eTqTBAiYhaRR+rapg6RwMFqn9Pcy2o6TwwQByyptthLRBJEZI2IXNnAGM641XtzsJqF3lk/g9kTel1TtS7/+x+weZooG96HAA89WrqmaS1bfeakPhNdbW4AZrrPVylGKRUP3AS8JSKdjt9JRKa6k0hCVlbWGQjz5Fbvy2FQlC+W7TOgxyTwCQHAVVJCwfz5/N5dMaTT6OYNUtM0rR7qU8X0u4i8KyIjRaR/5ase+6UDHap9jnIvq8kNHFe9pJRKd//cByzj2PaJym0+VkrFK6Xiw8PD6xFS0yoos7M9vYCbAreBLf+YxunCXxZAWRlLe5sYETmi+YLUNE2rp/o0Uvd1/3yx2jKFMRRHXdYDXUQkDiMx3IBRGjiGiHQHgoHV1ZYFA6VKqXIRCQOGA6/XI9ZmtW5/Li4Fw4vmQ2AHiBtdtS7/++/Jj/Alr7M/3YK7NVuMmqZp9VWf0Vwb9LivUsohItOABRjdXD9TSu0QkReBBKXUHPemNwDfHTesRw/gIxFxYZRyXq3e+6mlWrU3mzhLDoGHf4dRT4DJKKBVHDhA2YYNLBrrzfAo3b1V07SzQ30mDIoA/ga0V0pdIiLnAUOVUp+ebF/3A3Xzjlv27HGfn69hv1VAr5Mdv6VZvTeH+0PWIYUcM7RG/g+zUSZhYY8K/qyrlzRNO0vUpw3iC4xSQHv35z3ooTZOkFNczu6MAi4uXwwdR0FwDADK6aRg9myye3egMEB3b9U07exRnwQRppSaAbjAqDoCnHXvcu5Zuz+XoaZEAsoPHfPsQ8mqVTiOHGFxL+gTrru3app29qhPgigRkVCMhmlEZAhQ0KRRnYVW7c3mJutylFcgdL+0ann+rO+RoEB+bJfOyKiRzRihpmnaqalPgngUmAN0EpHfgf8ADzRpVGehbckpjJd1SK/rwOoNgCM3l6LFi8kZdT5Os+jurZqmnVXq04tpo4iMArphTBS0Wyllb/LIziJHCm30yluE1Wo/5tmHgh/ngN3Osj5mwi3hunurpmlnlVoThIhMrmVVVxFBKfV9E8V01lmzL4frzMsoC+2Jd/u+gDEwX/7MmXj16c3Pso2xkWN191ZN084qdZUgLnP/bAMM4+g4SRcCqwCdINz2b1vLFab9uAa+VrWsbNNmKvbuxf74PRRVJOrqJU3Tzjq1Jgil1B0AIvIrcJ5S6rD7czuMrq+aW1TKLOxYsfa+rmpZ/syZmHx8WNHdhTnZrLu3app21qlPI3WHyuTgdgSIbqJ4zjppWXmMtS/jYMSYqoH5nMXFFM6fj+Xi0Xx9YCbD2g/T3Vs1TTvr1GcspsUisoCjg+ldDyxqupDOLgdXz2SoFFMUf/TZh8Kf56HKyviiUzou5eLpwU83Y4SapmkNU59eTNPcDdaVnfg/Vkr90LRhnT189/5MFkFE9bukaln+zJmUxUQw27qdZ+KfJco/qhkj1DRNa5h6TTnq7rGkG6WP5yinc8Ea1viNYYzFuJS2XbuwbdvGrPGeDGk/lGu7XtvMQWqapjXMSdsgRGSyiCSJSIGIFIpIkYgUnongWrqyPUvxoYyCDhdVLcubOROHRVjT24MXhr2gu7ZqmnbWqk8J4nXgMqXUzqYO5mxTsOkHnMqLkF7jAHDZbGT/MIs1XeHekY/T3q/9SY6gaZrWctWnF9MRnRxq4HLhn7KQ5a7e9I6JAODAT9OxlNjIGHM+V3e5upkD1DRNOz31KUEkiMh0YDZQXrnwnH+S+tBGfCty2OxzG5f6euBSLvb85328g03cPeUtXbWkadpZrz4liACgFLgY4+nqy4BJ9Tm4iEwQkd0ikiwiT9aw/k0R2ex+7RGR/GrrprjbPpJEZEq9fpszSO2ciwMTpTFjAfh+6XvEJBUik8bRzl9XLWmadvarTzfXOxpyYBExA+8BFwFpwHoRmVN96lCl1CPVtn8A6Od+HwI8B8RjDDO+wb1vXkNiaQqOnXNZ5+xBt7gOpBSmsPfrf9NDYMjd+pkHTdNah7oG63sH9xwQbgrIBpYqpVbW49iDgGSl1D738b4DrgBqm1v6RoykADAeWKiUynXvuxCYwNGH9ZpXdjLW3CR+dU1hclQAz654iD9udeA5chgeERHNHZ2maVqjqKsEkVDDshDg7yIyXSn11kmOHQkcrPY5DRhc04YiEgPEcXRAwJr2jTzJ+c6c3T8DsELi6VO2AlmzkaBiRcQNNzdzYJqmaY2nrsH6vqxpuYh8iDGa61uNGMcNwEyl1ClNZSoiU4GpANHRZ3B4qF3z2GfpRGhEZ3bkLGb8VjOW8FD8LrjgzMWgaZrWxOrTSH0MpVRZPTdNBzpU+xzlXlaTGzi2+qhe+yqlPlZKxSul4sPDw+sZ1mkqzkQdXMvc8n707RBE7sEkeifZCbzqKsRSrwfTNU3TzgqnlCBExCIid2BU+ZzMeqCLiMSJiAdGEphTwzG7A8HA6mqLFwAXi0iwiARj9KBacCqxNpk9vyAofnH0p190MJHL92BSEHR1bfMraZqmnZ3qaqQu4thGaoAyYDnwh5MdWCnlEJFpGDd2M/CZUmqHiLwIJCilKpPFDcB3SilVbd9cEXkJI8kAvFjZYN3sds2j2Ks9ibYYerT1wH99AbnnR9EjJqa5I9M0TWtUdbVB+J/uwZVS84B5xy179rjPz9ey72fAZ6cbQ6OqKIF9S9nofyltnF441i8kvBAyLhvd3JFpmqY1ulNugzin7V0CDhs/lPWhb4cgSmZ+T74PhF00sbkj0zRNa3Q6QZyKXfNweQXxU34sQ/ydeK7dztI+QnRox+aOTNM0rdHpBFFfTgfsmU9mxCgcWOi7fQXiUqwfGESgZ2BzR6dpmtbo6t0vU0TaAF6Vn5VSqU0SUUt1cA2U5bHeaxgW5cRv0c8kdw/CJzauuSPTNE1rEvWZMOhyEUkC9mP0YDoAzG/iuFqeXT+D2ZMfi7tzRXkKziNHWNzXRIy/7r2kaVrrVJ8qppeAIcAepVQcMBZY06RRtTRKwa6fUXGjWJdezoT9azCHhbE4uoDogDP4BLemadoZVJ8EYVdK5QAmETEppZZijLJ67shMhPwUsqLG4ZmbReSezbgmjcFpFqL9dYLQNK11qk+CyBcRP+A34BsReRsoadqwWphdPwPCWusgJqSsRVBkjDkfgJgAXcWkaVrrVJ8EcQXGhEEPA78AezEmDTp37PoZouJZexgmpKzDd+RI9vkUAegqJk3TWq2TJgilVAnGwHmj3SO8fgJUNHVgLUZBOhzeDN0vxb7yN0JshQTfcD2phamEeIXg73HaD5xrmqa1SPXpxXQPMBP4yL0oEmN+6nPDQaM93hY9ml4bl1AWaAzrnVKYotsfNE1r1epTxXQ/MBwoBFBKJQFtmjKoFiU7GYAdKU76Ze7BMWESYrGQWpiqq5c0TWvV6pMgypVSVVVKImLhxFFeW6+cZAjsQPasOSgg9rabKLWXklmWqRuoNU1r1eqTIJaLyNOAt4hcBPwP+Klpw2pBcpJQQZ0I/e1Xtnc4nzadojlYZMyGqksQmqa1ZvVJEE8AWcA2jHkg5gF/acqgWgylIDuZosN++JYUkD5yAgAphSkA+ilqTdNatTrHYhIRM7BDKdUd+PeZCakFKc6EiiKyN2dyxDuIkAuNOadTi4xhqHQJQtO01qzOEoRSygnsFpEG3QlFZIKI7BaRZBF5spZtrhORRBHZISLfVlvuFJHN7tcJU5WeETlJVBSZKd+VxoKYwfSLDQOMEkSoVyi+Vt9mCUvTNO1MqM9orsHADhFZR7UnqJVSl9e1k7v08R5wEcYc1utFZI5SKrHaNl2Ap4DhSqk894ixlcqUUn3r/Zs0hewkig8ZA9j+FjuQ59sZzzykFqbqBmpNq4HdbictLQ2bzdbcoWjH8fLyIioqCqvVWu996pMgnmlgPIOAZKXUPgAR+Q7jqezEatvcA7ynlMoDUEplNvBcTSMnmZJMb3KC2hDeNRZPixkwqphGRI5o5uA0reVJS0vD39+f2NhYRKS5w9HclFLk5OSQlpZGXFz9pyioz5PUy6u/ACdwXT2OHQkcrPY5zb2suq5AVxH5XUTWiMiEauu8RCTBvfzKepyv0anMJEozPUgI6UTfDkEAlNhLyC7L1iUITauBzWYjNDRUJ4cWRkQIDQ095ZJdvSYMEpF+wE3AtRjzQsw65QhrP38XYDQQBawQkV5KqXwgRimVLiIdgSUisk0ptfe4uKYCUwGioxu/wbgscTcuOySEdua66GDAqF4C9FPUmlYLnRxapob8XWotQYhIVxF5TkR2Ae8AqYAopS5USr1bj2OnY4zhVCnKvay6NGCOUsqulNoP7MFIGCil0t0/9wHLgH7Hn0Ap9bFSKl4pFR8eHl6PkE6B005Jci4AW8I7c567/SGlyN3FVZcgNE1r5eqqYtoFjAEmKaVGKKXewaheqq/1QBcRiRMRD+AG4PjeSLMxSg+ISBhGldM+EQkWEc9qy4dzbNtF08s7QGmGlYq2oRR5+BIV7AMcLUF08O9Q196apmlnvboSxGTgMLBURP4tImOBepdRlFIOYBqwANgJzFBK7RCRF0WksgfUAiBHRBKBpcBj7smJegAJIrLFvfzV6r2fzgRX+g5KczxIj+lK2wAvvKxGA3VKYQptvNvgY/U5k+FomtbIJk6cSH5+fp3bPPvssyxatOiUj71s2TJWrVp1yvslJCTw4IMPnvJ+TaXWNgil1Gxgtoj4YvQ+ehhoIyIfAD8opX492cGVUvMwnryuvuzZau8V8Kj7VX2bVUCvev8WTaB0zUpwCevb9iE69Ggy0IP0adrZTSmFUop58+addNsXX3yxQedYtmwZfn5+DBs27IR1DocDi6XmW298fDzx8S1nws6TNlK754P4FvhWRIIxGqqfAE6aIM5mJRu2IybFIp/ODA6pliCKUrmww4XNGJmmnR1e+GkHiYcKG/WY57UP4LnLep50u3/+85989tlnANx9991ceeWVjB8/nsGDB7NhwwbmzZvHqFGjSEhIICwsjJdeeomvv/6a8PBwOnTowIABA/jTn/7E7bffzqRJk7jmmmuIjY1lypQp/PTTT9jtdv73v//RvXv3E8594MABPvzwQ8xmM19//TXvvPMOn376KV5eXmzatInhw4dzww038NBDD2Gz2fD29ubzzz+nW7duLFu2jDfeeIO5c+fy/PPPk5qayr59+0hNTeXhhx8+46WLevViquR+XuFj96tVK9l9GK/2Xhwsg+vcCaKooohcW64uQWhaC7ZhwwY+//xz1q5di1KKwYMHM2rUKJKSkvjyyy8ZMmTIMduvX7+eWbNmsWXLFux2O/3792fAgAE1HjssLIyNGzfy/vvv88Ybb/DJJ5+csE1sbCx//OMf8fPz409/+hMAn376KWlpaaxatQqz2UxhYSG//fYbFouFRYsW8fTTTzNr1omdQ3ft2sXSpUspKiqiW7du3Hvvvaf0oNvpOqUEca5w5OVRfqQC/wuNnkqVVUxVYzDpLq6adlL1+abfFFauXMlVV12Fr68xFM7kyZP57bffiImJOSE5APz+++9cccUVeHl54eXlxWWX1T6j8uTJkwEYMGAA33///SnFde2112I2G22ZBQUFTJkyhaSkJEQEu91e4z6XXnopnp6eeHp60qZNG44cOUJUVNQpnfd01Gc013NO6W9LjZ/dugEQHXJsDyZdgtC0s09lwjgdnp6eAJjNZhwOR4PP/8wzz3DhhReyfft2fvrpp1ofYKs8X0PPebp0gqhByfJFmCwuUiPPB44miMphvnUXV01ruUaOHMns2bMpLS2lpKSEH374gZEjR9a6/fDhw6tu0sXFxcydO/e0Y/D396eoqKjW9QUFBURGGgNLfPHFF6d9vqaiE0QNShI249Omgl2u9vh5Wgjx9QCMEkSETwTeFu9mjlDTtNr079+f22+/nUGDBjF48GDuvvtugoODa91+4MCBXH755fTu3ZtLLrmEXr16ERgYeFoxXHbZZfzwww/07duX33777YT1jz/+OE899RT9+vU746WCUyFGT9OzX3x8vEpISDjt41SkpbN33Dgi+hfyxEX/I63QyfyHjG8fN8+7GS+zF5+O//S0z6NprdHOnTvp0aNHc4dxyoqLi/Hz86O0tJQLLriAjz/+mP79+zd3WI2upr+PiGxQStXYt1Y3Uh+ndM1qAHw6h7I/z07ncL+qdamFqYyLGddcoWma1kSmTp1KYmIiNpuNKVOmtMrk0BA6QRynZPUazD6CR5eupO4uZUx3Y4qKgvIC8svz9TSjmtYKffvttyffqAaff/45b7/99jHLhg8fznvvvdcYYTU7nSCqUUpRsmYNvm1slAV2pMLhooPuwaRpWi3uuOMO7rjjjuYOo8noRupqyvck4czJwbdNKZlWo69xTGUPJj2Kq6Zp5xidIKqpbH/wjaggRYwuaNWfgRCEKP8z95CKpmlac9IJopqSVauxRgRh9XWyyxGBSSAy2OjSmlqUSlvftniaPU9yFE3TtNZBJwg3ZbdTun49vp0CwcOPnUU+tA/yxmo2LpEexVXTtHONThBuZdu24SotxbetA0I7kZJbRky1Yb5TClN0DyZNa0Wacj6IU/XFF18wbdq0Jj/PqdK9mNxKVq8GEXwCjkDoIA7uLOXinhEA5NvyKawo1CUITWsFzsR8EK2FThBupavX4NW9OxbbEiqCbiCnpILoEGNwLd2DSdMaYP6TkLGtcY/Zthdc8upJN2vO+SBcLhcdO3Zk8+bNBAUFAdClSxdWrlzJunXr+Otf/0pFRQWhoaF88803RERENOolakxNWsUkIhNEZLeIJIvIk7Vsc52IJIrIDhH5ttryKSKS5H5Naco4XaWllG7Zgk+froAi08MYjE+P4qppZ5/q80GsWbOGf//73+Tl5ZGUlMR9993Hjh07iIk5+mWv+nwQ8+fPp64heyrng7j33nt54403atzGZDJxxRVX8MMPPwCwdu1aYmJiiIiIYMSIEaxZs4ZNmzZxww038PrrrzfuL9/ImqwEISJm4D3gIiANWC8ic6rPLS0iXYCngOFKqTwRaeNeHgI8B8QDCtjg3jevKWIt3bAB7HZ8u4bBHkiV9kB5VRtESmEKJjHRwU+P4qpp9VaPb/pNoSXMB3H99dfz4osvcscdd/Ddd99x/fXXA5CWlsb111/P4cOHqaioIC4u7nR+1SbXlCWIQUCyUmqfUqoC+A5jbuvq7gHeq7zxK6Uy3cvHAwuVUrnudQuBCU0VaMmq1YjVik8bJwC77cbwGtWfom7n2w6r+czN5KRpWuM6k/NBDB06lOTkZLKyspg9e3ZVYnnggQeYNm0a27Zt46OPPqp1HoiWoikTRCRwsNrnNPey6roCXUXkdxFZIyITTmFfRGSqiCSISEJWVlaDAy1Zswbvfv0wFR0A/3YkFwhBPlYCvY2EkFKUotsfNO0s0RLmgxARrrrqKh599FF69OhBaGgocOw8EF9++eVpn6epNXcjtQXoAowGooAVItKrvjsrparmx46Pj2/QuOWOvDzKd+4k/KEHIed7CO1Mam5pVfuDUorUwlR6d+zdkMNrmnaGVZ8PAjil+SAiIiIaZT4IMKqZBg4ceMyEQM8//zzXXnstwcHBjBkzhv3795/2eZpUZZevxn4BQ4EF1T4/BTx13DYfAndU+7wYGAjcCHxUbflHwI11nW/AgAGqIRxFxSpv5kxl27tXqVdjlJrzkLrg9SXq/m82KKWUyi7NVud/cb76asdXDTq+pp1LEhMTmzuEBikqKlJKKVVSUqIGDBigNmzY0MwRNY2a/j5AgqrlvtqUVUzrgS4iEiciHsANwJzjtpmNUXpARMIwqpz2AQuAi0UkWESCgYvdyxqd2c+XoKuvxjMiEMrycIZ0Jj2vrKoEcaDwAKB7MGlaazZ16lT69u1L//79ufrqq/V8EG5NVsWklHKIyDSMG7sZ+EwptUNEXsTIWHM4mggSASfwmFIqB0BEXsJIMgAvKqVymypWAHKSAcj1jsHhUlU9mBamLMRqstI7TFcxaVprpeeDqFmTtkEopeYB845b9my19wp41P06ft/PgM+aMr5j5CQBkCrtgCN0CPHB5rDx096fGBc9jiCvoDMWiqZpZwc9H8S5IjsJTFb22EIAiAn1ZWHKQgorCrm669XNHJymadqZpxNEpZxkCOlISn4FVrPQNsCLWUmziPaPZmDbgc0dnaZp2hmnE0SlnGR3F9cSOgT7kFp0gA1HNjC5y2RMoi+TpmnnHn3nA3A5IXcfhBnPQHQI8WHWnllYxMIVnY9/+FvTNO3coBMEQH4qOCtQoZ1JySklKsTKnL1zuDD6QsK8w5o7Ok3TGsHdd99NYmLiyTeswfPPP181OF9jzxGxefPmeg09frxDhw5xzTXXNFocNWnuJ6lbBncX12K/WIpsRdg9t5JXkMfVXXTjtKY11GvrXmNX7q5GPWb3kO48MeiJBu37ySefNEoMjT1HxObNm0lISGDixIknrHM4HFgsNd+m27dvz8yZMxs1luPpEgRUJYgU93BPyWWLae/bnqHthzZnVJqmNcCBAwfo3r07N998Mz169OCaa66htLSU0aNHVw3l7efnxyOPPELPnj0ZO3YslWO57d27lwkTJjBgwABGjhzJrl0nJrjbb7+96sYcGxvLc889R//+/enVq1fV9iUlJdx5550MGjSIfv368eOPP9YYa0VFBc8++yzTp0+nb9++TJ8+neeff55bb72V4cOHc+utt3LgwAFGjhxJ//796d+/P6tWrar6Pc8//3zAmJFu8uTJTJgwgS5duvD44483yrXUJQgwurh6BbG/1Aux5pBUtIlpfafpxmlNOw0N/abfGHbv3s2nn37K8OHDufPOO3n//fePWV9SUkJ8fDxvvvkmL774Ii+88ALvvvsuU6dO5cMPP6RLly6sXbuW++67jyVLltR5rso5It5//33eeOMNPvnkE15++WXGjBnDZ599Rn5+PoMGDWLcuHEnjCjr4eHBiy++SEJCAu+++y5gVGclJiaycuVKvL29KS0tZeHChXh5eZGUlMSNN95Y45wVmzdvZtOmTXh6etKtWzceeOABOnQ4vSkKdIIA4yG50M6k5pVhDVqPWcxc2fnK5o5K07QG6tChA8OHDwfglltu4V//+tcx600mU9UcDbfccguTJ0+muLiYVatWce2111ZtV15eftJz1TRHxK+//sqcOXOq2i1sNhupqan06NGjXvFffvnleHt7A2C325k2bRqbN2/GbDazZ8+eGvcZO3Zs1SCD5513HikpKTpBNIqcvRB3AQeyC/EM3sDIqJFE+LbcaQA1TaubiNT5uabtXS4XQUFBbN68+ZTOVdMcEUopZs2aRbdu3U7pWJWqlzTefPNNIiIi2LJlCy6XCy8vrzrjOD6W06HrUCpKoDAdQjuzo2A1mIu4pkvT9gzQNK1ppaamsnr1asAYZ2nEiBHHrHe5XFXtCJXrAwICiIuL43//+x9g3OS3bNnSoPOPHz+ed955p3I0ajZt2lTrtv7+/hQVFdW6vqCggHbt2mEymfjqq69wOp0NiqkhdIJwlMPAuyF6KIecy/EkhOGRw5s7Kk3TTkO3bt1477336NGjB3l5edx7773HrPf19WXdunWcf/75LFmyhGefNYaI++abb/j000/p06cPPXv2rLVx+WSeeeYZ7HY7vXv3pmfPnjzzzDO1bnvhhReSmJhY1Uh9vPvuu48vv/ySPn36sGvXrkaZGa++pDLDne3i4+NVXZONn8z+/FQumz2Jfv7X8NXVz558B03TTrBz585617M3lQMHDjBp0iS2b99e6zZ+fn4UFxefwahahpr+PiKyQSkVX9P2ugTh9vUOo7g5uv2kZo5E0zStZdCN1IDD5WBB6k84S7rSu11sc4ejadppiI2NrbP0ADRL6WHBggU88cSxXX/j4uL44Ycfzngs9aUTBLAyfSUFFdnY8ycQ455JTtM0rTGNHz+e8ePHN3cYp6RJq5hEZIKI7BaRZBF5sob1t4tIlohsdr/urrbOWW358VOVNqpZe2bhJUFYbD0J9/c8+Q6apmnngCYrQYiIGXgPuAhIA9aLyByl1PGjZU1XSk2r4RBlSqm+TRVfpYySDFakr6CtuoTgEP+T9pfWNE07VzRlCWIQkKyU2qeUqgC+A1rc2NnBXsH8dfhfceQPJlpXL2maplVpygQRCRys9jnNvex4V4vIVhGZKSLVnwv3EpEEEVkjIlfWdAIRmereJqFysK1T5Wn2ZFLHSRzK9iY65Mz1L9Y0TWvpmrub609ArFKqN7AQ+LLauhh339ybgLdEpNPxOyulPlZKxSul4sPDwxscRFZxOWV2J9Eh3g0+hqZpLVtLnQ/iVC1btoxJk85Md/ym7MWUDlQvEUS5l1VRSuVU+/gJ8Hq1denun/tEZBnQD9jbFIEezC0FICZUlyA0rbFk/O1vlO9s3PkgPHt0p+3TTzdo35Y6H0RL1pQliPVAFxGJExEP4AbgmN5IItKu2sfLgZ3u5cEi4ul+HwYMBxqW+ushJcdIEB10G4SmnfXOpvkgAIYMGcKOHTuqPlfGuW7dOoYOHUq/fv0YNmwYu3fvbrRrVF9NVoJQSjlEZBqwADADnymldojIi0CCUmoO8KCIXA44gFzgdvfuPYCPRMSFkcReraH3U6NJzS1FBKKCdRWTpjWWhn7Tbwxny3wQANdffz0zZszghRde4PDhwxw+fJj4+HgKCwv57bffsFgsLFq0iKeffppZs2Y16nU6mSZ9UE4pNQ+Yd9yyZ6u9fwp4qob9VgG9mjK26lJzSmkb4IWX1XymTqlpWhM6m+aDuO6667j44ot54YUXmDFjRtU80wUFBUyZMoWkpCREBLvdfqqX4bTpJ6kxShC6i6umtR5n03wQkZGRhIaGsnXrVqZPn86HH34IGCPCXnjhhfzwww8cOHCA0aNHn1JcjaG5ezG1CCk6QWhaq3I2zQcBRjXT66+/TkFBAb179waMEkRkpPFkwBdffNGgOE7XOZ8gyiqcZBWVExOqE4SmtRZn03wQANdccw3fffcd1113XdWyxx9/nKeeeop+/fo1yuxwDXHOzweRU1zO8z8lcu2AKC7o2vBnKTRN0/NBtHSnOh/EOd8GEernyTs39mvuMDRN01qccz5BaJrWuuj5IBqPThCapjUqpZQeFbkGzT0fREOaE875RmpN0xqPl5cXOTk5DboZaU1HKUVOTg5eXl6ntJ8uQWia1miioqJIS0ujoaMra03Hy8uLqKioU9pHJwhN0xqN1WolLi6uucPQGomuYtI0TdNqpBOEpmmaViOdIDRN07QatZonqUUkC0g5jUOEAdmNFE5j07E1jI6tYXRsDXO2xhajlKpxGIlWkyBOl4gk1Pa4eXPTsTWMjq1hdGwN0xpj01VMmqZpWo10gtA0TdNqpBPEUR83dwB10LE1jI6tYXRsDdPqYtNtEJqmaVqNdAlC0zRNq5FOEJqmaVqNzvkEISITRGS3iCSLyJPNHU91InJARLaJyGYROfXp8ho/ns9EJFNEtldbFiIiC0Ukyf0zuIXE9byIpLuv3WYRmXim43LH0UFElopIoojsEJGH3MtbwnWrLbZmv3Yi4iUi60Rkizu2F9zL40Rkrfv/63QR8WhBsX0hIvurXbe+Zzq2ajGaRWSTiMx1f27YdVNKnbMvwAzsBToCHsAW4LzmjqtafAeAsOaOo1o8FwD9ge3Vlr0OPOl+/yTwWguJ63ngTy3gmrUD+rvf+wN7gPNayHWrLbZmv3aAAH7u91ZgLTAEmAHc4F7+IXBvC4rtC+Ca5v43547rUeBbYK77c4Ou27leghgEJCul9imlKoDvgCuaOaYWSym1Asg9bvEVwJfu918CV57JmKDWuFoEpdRhpdRG9/siYCcQScu4brXF1uyUoXLaN6v7pYAxwEz38ua6brXF1iKISBRwKfCJ+7PQwOt2rieISOBgtc9ptJD/IG4K+FVENojI1OYOphYRSqnD7vcZQERzBnOcaSKy1V0FdcarcI4nIrFAP4xvnC3quh0XG7SAa+euJtkMZAILMUr7+Uoph3uTZvv/enxsSqnK6/ay+7q9KSKezREb8BbwOOByfw6lgdftXE8QLd0IpVR/4BLgfhG5oLkDqosyyq8t5ZvUB0AnoC9wGPhHcwYjIn7ALOBhpVRh9XXNfd1qiK1FXDullFMp1ReIwijtd2+OOGpyfGwicj7wFEaMA4EQ4Inaj9A0RGQSkKmU2tAYxzvXE0Q60KHa5yj3shZBKZXu/pkJ/IDxn6SlOSIi7QDcPzObOR4AlFJH3P+JXcC/acZrJyJWjBvwN0qp792LW8R1qym2lnTt3PHkA0uBoUCQiFROdNbs/1+rxTbBXWWnlFLlwOc0z3UbDlwuIgcwqszHAG/TwOt2rieI9UAXdwu/B3ADMKeZYwJARHxFxL/yPXAxsL3uvZrFHGCK+/0U4MdmjKVK5c3X7Sqa6dq5638/BXYqpf5ZbVWzX7faYmsJ105EwkUkyP3eG7gIo41kKXCNe7Pmum41xbarWsIXjDr+M37dlFJPKaWilFKxGPezJUqpm2nodWvu1vbmfgETMXpv7AX+3NzxVIurI0avqi3AjpYQG/BfjCoHO0Y95l0Y9ZuLgSRgERDSQuL6CtgGbMW4Gbdrpms2AqP6aCuw2f2a2EKuW22xNfu1A3oDm9wxbAeedS/vCKwDkoH/AZ4tKLYl7uu2Hfgad0+n5noBoznai6lB100PtaFpmqbV6FyvYtI0TdNqoROEpmmaViOdIDRN07Qa6QShaZqm1UgnCE3TNK1GOkFo2ikQEWe10To3SyOOACwisdVHpNW05mY5+SaaplVTpowhFjSt1dMlCE1rBGLM3fG6GPN3rBORzu7lsSKyxD2A22IRiXYvjxCRH9xzCmwRkWHuQ5lF5N/ueQZ+dT+pq2nNQicITTs13sdVMV1fbV2BUqoX8C7GiJoA7wBfKqV6A98A/3Iv/xewXCnVB2Muix3u5V2A95RSPYF84Oom/W00rQ76SWpNOwUiUqyU8qth+QFgjFJqn3sAvAylVKiIZGMMVWF3Lz+slAoTkSwgShkDu1UeIxZj6Ogu7s9PAFal1F/PwK+maSfQJQhNazyqlvenorzaeye6nVBrRjpBaFrjub7az9Xu96swRtUEuBn4zf1+MXAvVE0+E3imgtS0+tLfTjTt1Hi7ZxKr9ItSqrKra7CIbMUoBdzoXvYA8LmIPAZkAXe4lz8EfCwid2GUFO7FGJFW01oM3QahaY3A3QYRr5TKbu5YNK2x6ComTdM0rUa6BKFpmqbVSJcgNE3TtBrpBKFpmqbVSCcITdM0rUY6QWiapmk10glC0zRNq9H/A++FgPEIFm4FAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "for history in histories:\n",
    "    plt.plot(history['train_auc_1'])\n",
    "    plt.plot(history['val_auc_1'])\n",
    "\n",
    "plt.title('Model Area Under Curve')\n",
    "plt.ylabel('Area Under Curve')\n",
    "plt.xlabel('Epoch')\n",
    "plt.legend(\n",
    "    ['origin_train', 'origin_val', 'pipeline_train', 'pipeline_val'], loc='lower right'\n",
    ")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到，两个模型的验证集 auc 均在 0.85 左右波动，使用流水线并行对此任务的训练精度影响不大，而训练时间由 0.76 分钟下降到 0.65 分钟。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结\n",
    "\n",
    "本篇示例介绍了隐语拆分学习中流水线并行的使用方法。与一般拆分学习的使用方法一样，只需在定义 SLModel 时指定 strategy='pipeline' 并设置 pipeline_num 即可。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
